Debiasing LLM Judges: Understanding and correcting AI Evaluation Bias

Evaluating the outputs of large language models (LLMs) accurately is crucial, but using LLMs as judges can introduce biases and imperfections. LLM judges can be systematically wrong, overvaluing fluency over factual correctness, missing subtle reasoning, and favoring answers similar to their own outputs. These biases can skew evaluation outcomes in predictable ways, making it essential to correct for them. To quantify the problem, a small set of gold-labeled examples can be used to audit the judge and measure its bias. A correction formula can be applied to debias the observed win rate, derived from measurement theory used in psychology, medicine, and machine learning. This formula assumes that judge errors are independent of the model's identity, but in practice, this assumption is often violated. The correction formula can still be biased if the judge prefers certain model types, highlighting the need to validate the judge's fairness across model types. Alternative approaches to addressing LLM judge bias include gold human labeling, judge ensembling, self-consistency, adjudication, training a meta-evaluator, and confident learning. It is essential to treat evaluators as models with limitations, biases, and parameters that must be understood, audited, and corrected to ensure integrity and transparency in model evaluation. By acknowledging and addressing these biases, we can improve the trustworthiness of our metrics and evaluations.