Vish Vogeti

AutoMixer: Checkpoint Artifacts as Automatic Data Mixers

In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these capabilities as the relationship between data and tasks is difficult to be modeled. In this work, we observe that checkp…