Representational Curvature Shapes Behavioral Uncertainty in Large Language Models

Abstract

In autoregressive large language models (LLMs), temporal straightening offers an account of how the next-token prediction objective shapes representations. Across layers, models progressively straighten the trajectory of input sequences in activation space, potentially facilitating extrapolation to the next token. However, a direct link between this geometry and token-level behavior has been missing. We provide such a link by relating contextual curvature—a geometric measure of how sharply the representation trajectory bends over recent context—to next-token entropy. Across model families (GPT-2 XL and Pythia-2.8B), contextual curvature is correlated with entropy, and this relationship emerges during training. Perturbation experiments provide causal evidence: reducing curvature through trajectory-aligned interventions selectively lowers entropy, while geometrically misaligned perturbations have no effect. Finally, explicitly regularizing representations to be straighter during training modestly reduces token-level entropy without degrading validation loss. These results identify trajectory curvature as a task-aligned representational feature that influences output uncertainty, suggesting that temporal straightening could be a functional principle shaping prediction in autoregressive language models.