QEDBench: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Santiago Gonzalez, Alireza Amiribavandpour, Peter Ye, Edward Zhang, Ruslans Aleksejevs, Todor Antić, Polina Baron, Sujeet Bhalerao
Abstract
As Large Language Models (LLMs) saturate elementary benchmarks, the research frontier has shifted from generation to the reliability of automated evaluation. We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic evaluation Alignment Gap when applied to upper-undergraduate to early graduate level mathematics. To quantify this, we introduce QEDBench, the first benchmark to systematically measure alignment with human experts on undergraduate-level math proofs by contrasting course-specific rubrics against expert common knowledge criteria. By deploying a dual-evaluation matrix ($7$ judges $\times$ $5$ solvers) against 1,000+ hours of human evaluation, we reveal that certain frontier evaluators like Claude 4.5 Opus exhibit significant positive bias (up to $+0.28$ mean score inflation), effectively "hallucinating rigor" in flawed proofs. Furthermore, we uncover a critical reasoning disparity: while Gemini 3.0 Pro achieves state-of-the-art performance (0.91 raw score), specialized reasoning models like o3-deep-research collapse in discrete domains, dropping to 42.1\% accuracy in Graph Theory. We release QEDBench as a public benchmark for evaluating and improving AI judges.
BibTeX
@inproceedings{
gonzalez2026qedbench,
title={{QEDB}ench: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs},
author={Santiago Gonzalez and Alireza Amiri Bavandpour and Peter Ye and Edward Zhang and Ruslans Aleksejevs and Todor Anti{\'c} and Polina Baron and Sujeet Bhalerao and Shubhrajit Bhattacharya and Zachary Burton and John Byrne and Hyungjun Choi and Nujhat Ahmed Disha and Kopp{\'a}ny Istv{\'a}n Encz and Yuchen Fang and Robert Joseph George and Ebrahim Ghorbani and Alan Goldfarb and Jing Guo and Meghal Gupta and Stefano Huber and Annika Kanckos and Minjung Kang and Hyun Jong Kim and Dino Lorenzini and Levi Lorenzo and Tianyi Mao and Giovanni Marzenta and Ariane M. Masuda and Lukas Mauth and Ana Mickovic and Andr{\'e}s Miniguano-Trujillo and Antoine Moulin and Wenqi Ni and Tomos Parry and Kevin Ren and Hossein Roodbarani and Mathieu Rundstr{\"o}m and Manjil Saikia and Detchat Samart and Rebecca Steiner and Connor Stewart and Dhara Thakkar and Jeffrey Tse and Vasiliki Velona and Yunhai Xiang and Sibel Yal{\c{c}}{\i}n and Jun Yan and Ji Zeng and Arman Cohan and Quanquan C. Liu},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=aOL0RPDfTQ}
}