2026
Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention
ICML 2026poster
Linear attention mitigates the quadratic complexity of softmax attention but suffers from a critical loss of expressiveness. We identify two primary causes: (1) The normalization operation cancels the query norm, which breaks the correlation between a query's norm and the spikiness (entropy) of the …