2026
Bits That Count: Quantifying and Predicting Capabilities of Language Models
ICML 2026poster
What and how do language models learn during training? When does learning elicit \textit{existing} knowledge, and when does it primarily teach \textit{new} capabilities? We find that the amount of generalizable information language models learn during training predicts the origins of their emergent …