OmniBench: Towards The Future of Universal Omni-Language Models
Yizhi LI, Ge Zhang, Yinghao Ma, Ruibin Yuan, King Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu
Abstract
Recent advancements in multimodal large language models (MLLMs) have focused on integrating multiple modalities, yet their ability to simultaneously process and reason across different inputs remains underexplored. We introduce OmniBench, a novel benchmark designed to evaluate models’ ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define language models capable of such tri-modal processing as omni-language models (OLMs). OmniBench features high-quality human annotations that require integrated understanding across all modalities. Our evaluation reveals that: i) open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts; and ii) most baseline models perform poorly (below 50% accuracy) even with textual alternatives to image/audio inputs. To address these limitations, we develop OmniInstruct, an 96K-sample instruction tuning dataset for training OLMs. We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance. Codes and data could be found at https://m-a-p.ai/OmniBench/.
BibTeX
@inproceedings{
li2025omnibench,
title={OmniBench: Towards The Future of Universal Omni-Language Models},
author={Yizhi LI and Ge Zhang and Yinghao Ma and Ruibin Yuan and King Zhu and Hangyu Guo and Yiming Liang and Jiaheng Liu and Zekun Moore Wang and Jian Yang and Siwei Wu and Xingwei Qu and Jinjie Shi and Xinyue Zhang and Zhenzhu Yang and Yidan WEN and Yanghai Wang and Shihao Li and Zhaoxiang Zhang and Ruibo Liu and Emmanouil Benetos and Wenhao Huang and Chenghua Lin},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025},
url={https://openreview.net/forum?id=SSF4qgsNYE}
}