Lior Belenki

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models

We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mixture of Data Experts (MDE). We use this approximation as a source of additional features in a regression model, trained…