When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

Abstract

Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 3.84B parameters, sparsity up to 90\%, and datasets from 1.3B to 41.6B tokens, trained for as many as 16 epochs. We find that: 1. **Sparse scaling in data-limited settings:** We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. **Delayed data saturation:** sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. **Resource trade-offs:** With fixed data, loss-optimal sparsity is moderate (~50%), while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity.