Kefei Tao

The Geometric Origin of Grokking: Accelerating Generalization via Active Structural Reorganization

Grokking, the phenomenon where models suddenly generalize long after overfitting training data, remains a puzzling challenge in neural network dynamics. Through mechanistic analysis, we find that this transition is fundamentally driven by a structural reorganization of token embeddings, with the ons…