Rotary Position Encodings for Graphs
Isaac Reid, Arijit Sehanobish, Cederik Höfs, Bruno Mlodozeniec, Leonhard Vulpius, Federico Barbero, Adrian Weller, Krzysztof Choromanski
Abstract
We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention.
BibTeX
@inproceedings{
reid2026rotary,
title={Rotary Position Encodings for Graphs},
author={Isaac Reid and Arijit Sehanobish and Cederik H{\"o}fs and Bruno Kacper Mlodozeniec and Leonhard Vulpius and Federico Barbero and Adrian Weller and Krzysztof Marcin Choromanski and Richard E. Turner and Petar Veli{\v{c}}kovi{\'c}},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=trn64znfNx}
}