MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu
Abstract
Recent advances in large language models (LLMs) have increased the demand for comprehensive benchmarks to evaluate their capabilities as human-like agents. Existing benchmarks, while useful, often focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. This lack of granularity makes it difficult to deeply discern where failures stem from. Additionally, setting up these environments requires considerable effort, and issues of unreliability and reproducibility sometimes arise, especially in interactive tasks. To address these limitations, we introduce the Massive Multitask Agent Understanding (MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need for complex environment setups. It evaluate models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics, and covering five essential capabilities: Understanding, Reasoning, Planning, Problem-solving, and Self-correction. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents. By testing 20 representative models on MMAU, we provide deep and insightful analyses. Ultimately, MMAU not only sheds light on the capabilities and limitations of LLM agents but also enhances the interpretability of their performance.
BibTeX
@inproceedings{yin-etal-2025-mmau,
title = "{MMAU}: A Holistic Benchmark of Agent Capabilities Across Diverse Domains",
author = "Yin, Guoli and
Bai, Haoping and
Ma, Shuang and
Nan, Feng and
Sun, Yanchao and
Xu, Zhaoyang and
Ma, Shen and
Lu, Jiarui and
Kong, Xiang and
Zhang, Aonan and
Yap, Dian Ang and
Zhang, Yizhe and
Ahnert, Karsten and
Kamath, Vik and
Berglund, Mathias and
Walsh, Dominic and
Gindele, Tobias and
Wiest, Juergen and
Lai, Zhengfeng and
Wang, Xiaoming Simon and
Shan, Jiulong and
Cao, Meng and
Pang, Ruoming and
Wang, Zirui",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-naacl.267/",
pages = "4737--4765",
ISBN = "979-8-89176-195-7"
}