2025
Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
ICML 2025poster
Despite the success of Transformers, handling longer contexts remains challenging due to the limited length generalization and quadratic complexity of self-attention, which often requires post-training with a larger attention window, significantly increasing computational and memory costs. In this p…