Can MLLMs Understand the Deep Implication Behind Chinese Images?
Chenhao Zhang, Xi Feng, Yuelin Bai, Xeron Du, Jinchang Hou, Kaixin Deng, Guangzeng Han, Qinrui Li
Abstract
As the capabilities of Multimodal Large Language Models (MLLMs) improve, the need for higher-order evaluation of them is increasing. However, there is a lack of work evaluating MLLM for higher-order perception and understanding of Chinese visual content. To address this, we introduce the CII-Bench, which aims to assess MLLMs’ such capabilities for Chinese images. To ensure the authenticity of the Chinese context, images in CII-Bench are sourced from the Chinese Internet and manually reviewed, with corresponding answers also manually crafted. Additionally, CII-Bench incorporates images that represent Chinese traditional culture, such as famous Chinese traditional paintings, which can deeply reflect the model’s understanding of Chinese traditional culture. Through experiments on multiple MLLMs using CII-Bench, significant findings emerged. There is a large gap between MLLMs and humans in performance. The highest MLLM accuracy is 64.4%, while the human average is 78.2% and the peak is 81.0%. MLLMs perform poorly on traditional culture images, indicating limitations in understanding high-level semantics and lacking a deep knowledge base of Chinese traditional culture. Moreover, most models have higher accuracy when image emotion hints are added to the prompts. We believe CII-Bench will help MLLMs better understand Chinese semantics and specific images, and move forward the development of expert artificial general intelligence (AGI). Our project is publicly available at https://cii-bench.github.io.
BibTeX
@inproceedings{zhang-etal-2025-mllms,
title = "Can {MLLM}s Understand the Deep Implication Behind {C}hinese Images?",
author = "Zhang, Chenhao and
Feng, Xi and
Bai, Yuelin and
Du, Xeron and
Hou, Jinchang and
Deng, Kaixin and
Han, Guangzeng and
Li, Qinrui and
Wang, Bingli and
Liu, Jiaheng and
Qu, Xingwei and
Zhang, Yifei and
Zhao, Qixuan and
Liang, Yiming and
Liu, Ziqiang and
Fang, Feiteng and
Yang, Min and
Huang, Wenhao and
Lin, Chenghua and
Zhang, Ge and
Ni, Shiwen",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.700/",
doi = "10.18653/v1/2025.acl-long.700",
pages = "14369--14402",
ISBN = "979-8-89176-251-0"
}