Amaru Cuba Gyllensten — accepted AI-conference papers

2025

SWEb: A Large Web Dataset for the Scandinavian Languages

Tobias Norlund, Tim Isbister, Amaru Cuba Gyllensten, Paul Gabriel dos Santos, Danila Petrelli, Ariel Ekgren +1

ICLR 2025poster

This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces comple…

Cited by 0 Source ↗PDF ↗Scholar ↗

2024

GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages

Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou +3

COLING 2024main

This paper details the process of developing the first native large generative language model for the North Germanic languages, GPT-SW3. We cover all parts of the development process, from data collection and processing, training configuration and instruction finetuning, to evaluation, applications,…

Cited by 4 Source ↗PDF ↗code ↗Scholar ↗

2021

Semantic Re-tuning with Contrastive Tension

Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, Magnus Sahlgren

ICLR 2021poster

Extracting semantically useful natural language sentence representations from pre-trained deep neural networks such as Transformers remains a challenge. We first demonstrate that pre-training objectives impose a significant task bias onto the final layers of models with a layer-wise survey of the Se…

Cited by 98 Source ↗PDF ↗Scholar ↗