2026
Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws
ICML 2026spotlight
Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for generalization remain poorly understood. We study empirical risk m…