MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack
Abstract
We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time.We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (October 2024) achieving just a 41.4% average accuracy.
BibTeX
@inproceedings{deshpande-etal-2025-multichallenge,
title = "{M}ulti{C}hallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier {LLM}s",
author = "Deshpande, Kaustubh and
Sirdeshmukh, Ved and
Mols, Johannes Baptist and
Jin, Lifeng and
Hernandez-Cardona, Ed-Yeremai and
Lee, Dean and
Kritz, Jeremy and
Primack, Willow E. and
Yue, Summer and
Xing, Chen",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.958/",
doi = "10.18653/v1/2025.findings-acl.958",
pages = "18632--18702",
ISBN = "979-8-89176-256-5"
}