See the World, Discover Knowledge: A Chinese Factuality Evaluation for Large Vision Language Models
Jihao Gu, Yingyao Wang, Pi Bu, Chen Wang, Ziming Wang, Tengtao Song, Donglai Wei, Jiale Yuan
Abstract
The evaluation of factual accuracy in large vision language models (LVLMs) has lagged behind their rapid development, making it challenging to fully reflect these models’ knowledge capacity and reliability. In this paper, we introduce the first factuality-based visual question-answering benchmark in Chinese, named ChineseSimpleVQA, aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the Chinese language, diverse knowledge types, a multi-hop question construction, high-quality data, static consistency, and easy-to-evaluate through short answers. Moreover, we contribute a rigorous data construction pipeline and decouple the visual factuality into two parts: seeing the world (i.e., object recognition) and discovering knowledge. This decoupling allows us to analyze the capability boundaries and execution mechanisms of LVLMs. Subsequently, we evaluate 34 advanced open-source and closed-source models, revealing critical performance gaps within this field.
BibTeX
@inproceedings{gu-etal-2025-see,
title = "See the World, Discover Knowledge: A {C}hinese Factuality Evaluation for Large Vision Language Models",
author = "Gu, Jihao and
Wang, Yingyao and
Bu, Pi and
Wang, Chen and
Wang, Ziming and
Song, Tengtao and
Wei, Donglai and
Yuan, Jiale and
Zhao, Yingxiu and
He, Yancheng and
Li, Shilong and
Liu, Jiaheng and
Cao, Meng and
Song, Jun and
Tan, Yingshui and
Li, Xiang and
Su, Wenbo and
Zhu, Xiaoyong and
Zheng, Bo",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.844/",
doi = "10.18653/v1/2025.findings-acl.844",
pages = "16422--16447",
ISBN = "979-8-89176-256-5"
}