Preprint: Rethinking LLM-as-a-Judge: Biases and Limitations in Evaluating Chain-of-Thought Outputs

Abstract

The widespread adoption of Large Language Models (LLMs) has created a growing demand for scalable and reliable evaluation methodologies. While human evaluation remains the gold standard, it is costly, time-consuming, and often requires domain expertise. Consequently, LLM-as-a-Judge approaches have emerged as a popular alternative and are increasingly used to assess model outputs across a wide range of tasks.
The recent development of o1-like language models, capable of generating long chain-of-thought traces before producing a final answer, has introduced new challenges for evaluation. Assessing the quality of a reasoning process requires more than verifying the correctness of the final output, raising the question of whether LLM judges can reliably evaluate complex reasoning traces. In this work, we conduct a controlled study of LLM-as-a-Judge for evaluating mathematical reasoning, focusing on how changes in the presentation of reasoning traces affect model judgments.
Our experiments investigate whether enabling additional reasoning makes LLM judges more robust to changes in the presentation of reasoning traces. Using PRM800K, we construct controlled pairs consisting of the same solution in an uncorrupted form and a corrupted form containing a controlled number of steps with negative human annotations, and compare judgments with and without reasoning. We then perturb the presentation of these solution pairs by changing their order and their surface formulation, allowing us to measure whether judgments remain consistent when the underlying reasoning quality is unchanged.
These findings challenge the reliability of existing LLM-as-a-Judge methodologies for reasoning evaluation and highlight the need for more rigorous and robust evaluation frameworks for reasoning-capable language models.