From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs?
Geng Zhang, Yizhou Ying, Sihang Jiang, Jiaqing Liang, Guanglei Yue, Yifei Fu, Hailin Hu, Yanghua Xiao
Abstract
Despite the rapid development of large language models (LLMs), existing benchmark datasets often focus on low-level cognitive tasks, such as factual recall and basic comprehension, while providing limited coverage of higher-level reasoning skills, including analysis, evaluation, and creation. In this work, we systematically assess the cognitive depth of popular LLM benchmarks using Bloom’s Taxonomy to evaluate both the cognitive and knowledge dimensions.Our analysis reveals a pronounced imbalance: most datasets concentrate on “Remembering” and “Understanding”, with metacognitive and creative reasoning largely underrepresented. We also find that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. These findings highlight the importance of future benchmarks incorporating metacognitive evaluations to more accurately assess and enhance model performance.
BibTeX
@inproceedings{emnlp2025_fromrememberingt,
title = {From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs?},
author = {Geng Zhang and Yizhou Ying and Sihang Jiang and Jiaqing Liang and Guanglei Yue and Yifei Fu and Hailin Hu and Yanghua Xiao},
booktitle = {EMNLP 2025},
year = {2025}
}