2026
Focus and Dilution: The Multi-stage Learning Process of Attention
ICML 2026oral
Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identify a recurrent focus–dilution cycle in attention learning and provide a rigorous explanation in a one-layer Transformer s…