2025
Learning Dynamics in Continual Pre-Training for Large Language Models
ICML 2025oral
Continual Pre-Training (CPT) has become a popular and effective method to apply strong foundation models to specific downstream tasks. In this work, we explore the **learning dynamics** throughout the CPT process for large language models (LLMs). We specifically focus on how general and downstream…