2025
Constrained Belief Updates Explain Geometric Structures in Transformer Representations
ICML 2025poster
What computational structures emerge in transformers trained on next-token prediction? In this work, we provide evidence that transformers implement constrained Bayesian belief updating---a parallelized version of partial Bayesian inference shaped by architectural constraints. We integrate the model…