2026
Procedural Pretraining: Warming Up Language Models with Abstract Data
ICML 2026oral
Pretraining directly on web-scale corpora is the de facto paradigm for building language models. We study an alternative setting where the model is initially exposed to abstract structured data, as a means to ease the subsequent acquisition of rich semantic knowledge, much like humans learn simple l…