CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
David Orlando Romero Mogrovejo, Chenyang Lyu, Haryo Akbarianto Wibowo, Santiago Góngora, Aishik Mandal, Sukannya Purkayastha, Jesus-German Ortiz-Barajas, Emilio Villa Cueva
Abstract
Visual Question Answering~(VQA) is an important task in multimodal AI, which requires models to understand and reason on knowledge present in visual and textual data. However, most of the current VQA datasets and models are primarily focused on English and a few major world languages, with images that are Western-centric. While recent efforts have tried to increase the number of languages covered on VQA datasets, they still lack diversity in low-resource languages. More importantly, some datasets extend the text to other languages, either via translation or some other approaches, but usually keep the same images, resulting in narrow cultural representation. To address these limitations, we create CVQA, a new Culturally-diverse Multilingual Visual Question Answering benchmark dataset, designed to cover a rich set of languages and regions, where we engage native speakers and cultural experts in the data collection process. CVQA includes culturally-driven images and questions from across 28 countries in four continents, covering 26 languages with 11 scripts, providing a total of 9k questions. We benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and we show that the dataset is challenging for the current state-of-the-art models. This benchmark will serve as a probing evaluation suite for assessing the cultural bias of multimodal models and hopefully encourage more research efforts towards increasing cultural awareness and linguistic diversity in this field.
BibTeX
@inproceedings{
mogrovejo2024cvqa,
title={{CVQA}: Culturally-diverse Multilingual Visual Question Answering Benchmark},
author={David Orlando Romero Mogrovejo and Chenyang Lyu and Haryo Akbarianto Wibowo and Santiago G{\'o}ngora and Aishik Mandal and Sukannya Purkayastha and Jesus-German Ortiz-Barajas and Emilio Villa Cueva and Jinheon Baek and Soyeong Jeong and Injy Hamed and Zheng Xin Yong and Zheng Wei Lim and Paula M{\'o}nica Silva and Jocelyn Dunstan and M{\'e}lanie Jouitteau and David LE MEUR and Joan Nwatu and Ganzorig Batnasan and Munkh-Erdene Otgonbold and Munkhjargal Gochoo and Guido Ivetta and Luciana Benotti and Laura Alonso Alemany and Hern{\'a}n Maina and Jiahui Geng and Tiago Timponi Torrent and Frederico Belcavello and Marcelo Viridiano and Jan Christian Blaise Cruz and Dan John Velasco and Oana Ignat and Zara Burzo and Chenxi Whitehouse and Artem Abzaliev and Teresa Clifford and Gr{\'a}inne Caulfield and Teresa Lynn and Christian Salamea-Palacios and Vladimir Araujo and Yova Kementchedjhieva and Mihail Minkov Mihaylov and Israel Abebe Azime and Henok Biadglign Ademtew and Bontu Fufa Balcha and Naome A Etori and David Ifeoluwa Adelani and Rada Mihalcea and Atnafu Lambebo Tonja and Maria Camila Buitrago Cabrera and Gisela Vallejo and Holy Lovenia and Ruochen Zhang and Marcos Estecha-Garitagoitia and Mario Rodr{\'\i}guez-Cantelar and Toqeer Ehsan and Rendi Chevi and Muhammad Farid Adilazuarda and Ryandito Diandaru and Samuel Cahyawijaya and Fajri Koto and Tatsuki Kuribayashi and Haiyue Song and Aditya Nanda Kishore Khandavally and Thanmay Jayakumar and Raj Dabre and Mohamed Fazli Mohamed Imam and Kumaranage Ravindu Yasas Nagasinghe and Alina Dragonetti and Luis Fernando D'Haro and Olivier NIYOMUGISHA and Jay Gala and Pranjal A Chitale and Fauzan Farooqui and Thamar Solorio and Alham Fikri Aji},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024},
url={https://openreview.net/forum?id=E18kRXTGmV}
}