2026
Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding
ICLR 2026poster
So far, expensive finetuning beyond the pretraining sequence length has been a prerequisite to effectively extend the context of language models (LM). In this work, we break this key bottleneck by ***Dro**pping the **P**ositional **E**mbeddings of LMs after training (DroPE)*. Our simple method is mo…