2025
DataDecide: How to Predict Best Pretraining Data with Small Experiments
ICML 2025poster
Because large language models are expensive to pretrain on different datasets, using smaller-scale experiments to decide on data is crucial for reducing costs. Which benchmarks and methods of making decisions from observed performance at small scale most accurately predict the datasets that yield th…