EvalAssist: LLM-as-a-Judge Simplified
Michael Desmond, Zahra Ashktorab, Werner Geyer, Elizabeth M. Daly, Martín Santillán Cooper, Qian Pan, Rahul Nair, Nico Wagner
Abstract
We present EvalAssist, a framework that simplifies the LLM- as-a-judge workflow. The system provides an online criteria development environment, where users can interactively build, test, and share custom evaluation criteria in a structured and portable format. A library of LLM based evaluators is made available that incorporates various algorithmic innovations such as token-probability based judgement, positional bias checking, and certainty estimation that help to engender trust in the evaluation process. We have computed extensive benchmarks and also deployed the system internally in our organization with several hundreds of users.
BibTeX
@article{Desmond_Ashktorab_Geyer_Daly_Santillán Cooper_Pan_Nair_Wagner_Pedapati_2025, title={EvalAssist: LLM-as-a-Judge Simplified}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/35351}, DOI={10.1609/aaai.v39i28.35351}, abstractNote={We present EvalAssist, a framework that simplifies the LLM- as-a-judge workflow. The system provides an online criteria development environment, where users can interactively build, test, and share custom evaluation criteria in a structured and portable format. A library of LLM based evaluators is made available that incorporates various algorithmic innovations such as token-probability based judgement, positional bias checking, and certainty estimation that help to engender trust in the evaluation process. We have computed extensive benchmarks and also deployed the system internally in our organization with several hundreds of users.}, number={28}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Desmond, Michael and Ashktorab, Zahra and Geyer, Werner and Daly, Elizabeth M. and Santillán Cooper, Martín and Pan, Qian and Nair, Rahul and Wagner, Nico and Pedapati, Tejaswini}, year={2025}, month={Apr.}, pages={29637-29639} }