2026
Incremental Learning of Sparse Attention Patterns in Transformers
ICML 2026poster
This paper studies simple transformers on a high-order Markov chain, where the model must incorporate knowledge from multiple past positions, each with different statistical importance. We show that transformers learn the task incrementally, with each stage induced by the acquisition or copying of i…