Tansheng Zhu

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

Rotary Position Embeddings (RoPE) are widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads. Using simplified retrieval tasks and length generalization scenarios, we show—both empirica…