2024
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
ACL 2024long
Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to…