LIME: Less Is More for MLLM Evaluation
King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shuyue Guo, Tianyu Zheng
Abstract
Multimodal Large Language Models (MLLMs) are measured on numerous benchmarks like image captioning, visual question answer, and reasoning. However, these benchmarks often include overly simple or uninformative samples, making it difficult to effectively distinguish the performance of different MLLMs. Additionally, evaluating models across many benchmarks creates a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated using a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding. Our experiments show that LIME reduces the number of samples by 76% and evaluation time by 77%, while it can more effectively distinguish different models’ abilities. Notably, we find that traditional automatic metrics like CIDEr are insufficient for evaluating MLLMs’ captioning performance, and excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://anonymous.4open.science/r/LIME-49CD
BibTeX
@inproceedings{zhu-etal-2025-lime,
title = "{LIME}: Less Is More for {MLLM} Evaluation",
author = "Zhu, King and
Zang, Qianbo and
Jia, Shian and
Wu, Siwei and
Fang, Feiteng and
Li, Yizhi and
Guo, Shuyue and
Zheng, Tianyu and
Guo, Jiawei and
Li, Bo and
Wu, Haoning and
Qu, Xingwei and
Yang, Jian and
Liu, Ruibo and
Yue, Xiang and
Liu, Jiaheng and
Lin, Chenghua and
Alinejad-Rokny, Hamid and
Yang, Min and
Ni, Shiwen and
Huang, Wenhao and
Zhang, Ge",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.474/",
doi = "10.18653/v1/2025.findings-acl.474",
pages = "9086--9121",
ISBN = "979-8-89176-256-5"
}