2025
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
ACL 2025long
Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show h…