ICLR 2025poster0 citations
SWEb: A Large Web Dataset for the Scandinavian Languages
Tobias Norlund, Tim Isbister, Amaru Cuba Gyllensten, Paul Gabriel dos Santos, Danila Petrelli, Ariel Ekgren, Magnus Sahlgren
Abstract
This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces complexity in comparison with rule-based approaches. We also introduce a new cloze-style benchmark for evaluating language models in Swedish, and use this test to compare models trained on the SWEb data to models trained on FineWeb, with competitive results. All data, models and code are shared openly.
datasetpre-trainingswedishdanishnorwegianicelandic
BibTeX
@inproceedings{
norlund2025sweb,
title={{SWE}b: A Large Web Dataset for the Scandinavian Languages},
author={Tobias Norlund and Tim Isbister and Amaru Cuba Gyllensten and Paul Gabriel dos Santos and Danila Petrelli and Ariel Ekgren and Magnus Sahlgren},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=vhPE3PtTgC}
}