2025
Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
ICML 2025poster
Attention layers are the core component of transformers, the current state-of-the-art neural network architecture. Alternatives to softmax-based attention are being explored due to its tendency to hinder effective information flow. Even *at initialisation*, it remains poorly understood why the propa…