2026
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
ICLR 2026poster
The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora. We introduce two openly licensed pre-training datasets, released under the Llama 3.3 Community License, that significantly enhance…