Mumin Jia

Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention

Human cognition is punctuated by abrupt, spontaneous shifts between topics—driven by emotional, contextual, or associative cues—a phenomenon known as spontaneous thought in neuroscience. In contrast, self-attention-based models rely on structured patterns over their inputs to predict each next token…