2026
Transformers learn factored representations
Adam Shai, Loren Amdahl-Culleton, Casper Christensen, Henry R Bigelow, Fernando Rosas, Alexander Boyd +3
ICML 2026poster
Transformers pretrained via next token prediction learn to factor their world into parts, representing these factors in orthogonal subspaces of the residual stream. We formalize two representational hypotheses: (1) a representation in the product space of all factors, whose dimension grows exponenti…