2024
Scaling Parameter-Constrained Language Models with Quality Data
EMNLP 2024industry
Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization.In this paper, we extend the conventional understanding of scaling la…