Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific LLMs
Yizhou Ying, Geng Zhang, Cui Danxin, Chengyu Du, Guanglei Yue, Sihang Jiang, Jiaqing Liang, Yifei Fu
Abstract
Data efficiency is crucial in domain-specific continual pre-training (CPT) of large language models (LLMs), especially under resource constraints. Aiming for “small data, big impact,” this work addresses the limitations of existing domain-specific data selection strategies, which often rely on scarce labeled data or computationally expensive LLMs. We introduce CDF Sampling with Grammatical Complexity (CDF-GC), an annotation-independent, efficient and interpretable data selection framework for CPT. Our approach comprehensively evaluates grammatical complexity using lexical diversity and syntactic complexity, and employs a cumulative distribution function (CDF)-based sampling strategy to balance complexity and diversity. To validate the effectiveness of CDF-GC, we conducted experiments on a financial dataset. The results demonstrate that CDF-GC significantly outperforms baselines, achieving 2.0% improvement in financial QA at the same selection ratio and even surpassing full-data training by 1.7% using only 20% of the data.
BibTeX
@inproceedings{emnlp2025_dataefficientsel,
title = {Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific LLMs},
author = {Yizhou Ying and Geng Zhang and Cui Danxin and Chengyu Du and Guanglei Yue and Sihang Jiang and Jiaqing Liang and Yifei Fu and Hailin Hu and Yanghua Xiao},
booktitle = {EMNLP 2025},
year = {2025}
}