Maria Ios Glarou

GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining

The performance of large language models (LLMs) across diverse downstream applications is fundamentally governed by the quality and composition of their pretraining corpora. Existing domain reweighting algorithms primarily optimize data mixtures for a single target task, thereby resulting in models…