2025
What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers
NeurIPS 2025poster
Training Transformers on algorithmic tasks frequently demonstrates an intriguing *abrupt learning* phenomenon: an extended performance plateau followed by a sudden, sharp improvement. This work investigates the underlying mechanisms for such dynamics, primarily in shallow Transformers. We reveal tha…