Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models
Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo
Abstract
New LLM benchmarks are important to align with the rapid development of Large Language Models (LLMs). In this work, we present Chinese SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality ability of LLMs to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6 major topics with 99 diverse subtopics. Second, we conduct a comprehensive quality control process to achieve high-quality questions and answers, where the reference answers are static and cannot be changed over time. Third, following SimpleQA, the questions and answers are very short, and the grading process is easy-to-evaluate. Based on Chinese SimpleQA, we perform a comprehensive evaluation of the factuality abilities of existing LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to better understand the Chinese factuality abilities of their models and facilitate the growth of LLMs.
BibTeX
@inproceedings{he-etal-2025-chinese,
title = "{C}hinese {S}imple{QA}: A {C}hinese Factuality Evaluation for Large Language Models",
author = "He, Yancheng and
Li, Shilong and
Liu, Jiaheng and
Tan, Yingshui and
Wang, Weixun and
Huang, Hui and
Bu, Xingyuan and
Guo, Hangyu and
Hu, Chengwei and
Zheng, Boren and
Lin, Zhuoran and
Sun, Dekai and
Zheng, Zhicheng and
Su, Wenbo and
Zheng, Bo",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.941/",
doi = "10.18653/v1/2025.acl-long.941",
pages = "19182--19208",
ISBN = "979-8-89176-251-0"
}