O Duranthon

Statistical Advantage of Softmax Attention: Insights from Single-Location Regression

Large language models rely on attention mechanisms with a softmax activation. Yet the dominance of softmax over alternatives (e.g., component-wise or linear) remains poorly understood, and many theoretical works have focused on the easier-to-analyze linearized attention. In this work, we address thi…