yifan shang

Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm

Continual Pre-Training (CPT) is essential for enabling Language Models (LMs) to integrate new factual knowledge without erasing old. While classical CPT techniques like data replay have become the standard paradigm, the mechanisms underlying how LMs acquire and retain facts over time, termed as cont…