Random Scaling of Emergence Capabilities

Abstract

Language models famously improve under a smooth scaling law, but some specific capabilities exhibit sudden breakthroughs in performance. Advocates of "emergence" view breakthroughs as unlocked capabilities, but others attribute them to metric thresholding effects. We propose that breakthroughs are instead driven by continuous changes in the *probability distribution* of training outcomes when performance is bimodally distributed across random seeds. we show that different random seeds can produce *either* smooth *or* emergent scaling trends in synthetic length generalization tasks, multiple choice question answering, and grammatical generalization. We reveal that sharp breakthroughs in metrics are produced by underlying continuous changes in their distribution across seeds.