Revisiting Multimodal Positional Encoding in Vision–Language Models
Jie Huang, Xuejing Liu, Sibo Song, RuiBing Hou, Hong Chang, Junyang Lin, Shuai Bai
Abstract
Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors—ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding.
BibTeX
@inproceedings{
huang2026revisiting,
title={Revisiting Multimodal Positional Encoding in Vision{\textendash}Language Models},
author={Jie Huang and Xuejing Liu and Sibo Song and RuiBing Hou and Hong Chang and Junyang Lin and Shuai Bai},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=sCCF4ygDAw}
}