AIConfPaper
← Search

Mikel Artetxe

30 accepted papers

2025

BOUQuET : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation

Pierre Andrews, Mikel Artetxe, Mariano Coria Meglioli, Marta R. Costa-juss{\`a}, Joe Chuang, David Dale +13
EMNLP 2025

BOUQuET is a multi-way, multicentric and multi-register/domain dataset and benchmark, and a broader collaborative initiative. This dataset is handcrafted in 8 non-English languages (i.e. Egyptian Arabic and Modern Standard Arabic, French, German, Hindi, Indonesian, Mandarin Chinese, Russian, and Spa

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation

Ahmed Elhady, Eneko Agirre, Mikel Artetxe
ACL 2025long

Continued pretraining (CPT) is a popular approach to adapt existing large language models (LLMs) to new languages. When doing so, it is common practice to include a portion of English data in the mixture, but its role has not been carefully studied to date. In this work, we show that including Engli…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque

Oscar Sainz, Naiara Perez, Julen Etxaniz, Joseba Fernandez de Landa, Itziar Aldabe, Iker Garc{\'i}a-Ferrero +6
EMNLP 2025

Instructing language models with user intent requires large instruction datasets, which are only available for a limited set of languages. In this paper, we explore alternatives to conventional instruction adaptation pipelines in low-resource scenarios. We assume a realistic scenario for low-resourc

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

Linguini: A benchmark for language-agnostic linguistic reasoning

Eduardo Sánchez, Belen Alastruey, Christophe Ropers, Arina Turkatenko, Pontus Stenetorp, Mikel Artetxe +1
NeurIPS 2025poster

We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages, extracted from the International Linguistic Oly…

Cited by 0Source ↗code ↗Scholar ↗
2025

WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging

Ahmed Elhady, Eneko Agirre, Mikel Artetxe
ACL 2025short

We introduce WiCkeD, a simple method to increase the complexity of existing multiple-choice benchmarks by randomly replacing a choice with “None of the above”, a method often used in educational tests. We show that WiCkeD can be automatically applied to any existing benchmark, making it more challen…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

BertaQA: How Much Do Language Models Know About Local Culture?

Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, Mikel Artetxe
NeurIPS 2024poster

Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how well these models perform on topics relevant to other cultures, whose presence on the web is not that prominent. To add…

Cited by 5Source ↗PDF ↗code ↗Scholar ↗
2024

Do Multilingual Language Models Think Better in English?

Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, Mikel Artetxe
NAACL 2024short

Translate-test is a popular technique to improve the performance of multilingual language models. This approach works by translating the input into English using an external machine translation system before running inference. However, these improvements can be attributed to the use of a separate tr…

Cited by 54Source ↗PDF ↗code ↗Scholar ↗
2024

Improving Factuality in Clinical Abstractive Multi-Document Summarization by Guided Continued Pre-training

Ahmed Elhady, Khaled Elsayed, Eneko Agirre, Mikel Artetxe
NAACL 2024short

Factual accuracy is an important property of neural abstractive summarization models, especially in fact-critical domains such as the clinical literature. In this work, we introduce a guided continued pre-training stage for encoder-decoder models that improves their understanding of the factual attr…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

Latxa: An Open Language Model and Evaluation Suite for Basque

Julen Etxaniz, Oscar Sainz, Naiara Miguel, Itziar Aldabe, German Rigau, Eneko Agirre +3
ACL 2024long

We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the scarcity of high-quality benchmarks for Basque, we furth…

Cited by 21Source ↗PDF ↗code ↗Scholar ↗
2024

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa +4
ACL 2024long

We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in high-, medium-, and low-resource…

Cited by 89Source ↗PDF ↗code ↗Scholar ↗
2023

CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models

Aitor Ormazabal, Mikel Artetxe, Eneko Agirre
EMNLP 2023long main

Methods for adapting language models (LMs) to new tasks and domains have traditionally assumed white-box access to the model, and work by modifying its parameters. However, this is incompatible with a recent trend in the field, where the highest quality models are only available as black-boxes throu…

Cited by 0Source ↗Scholar ↗
2023

Improving Language Plasticity via Pretraining with Active Forgetting

Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebastian Riedel +1
NeurIPS 2023poster

Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible. While prior work has shown it possible to…

Cited by 23Source ↗PDF ↗code ↗Scholar ↗
2023

Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training

Kelly Marchisio, Patrick Lewis, Yihong Chen, Mikel Artetxe
ACL 2023findings

Prior work shows that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body frozen. Despite learning a small subset of parameters, this approach is not compute-efficient, as training the new embeddin…

Cited by 17Source ↗PDF ↗code ↗Scholar ↗
2023

Revisiting Machine Translation for Cross-lingual Classification

Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, Angela Fan, Luke Zettlemoyer
EMNLP 2023long main

Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (translate-test), or translating the training set into the target languages and finetuning a multilingual model (translate-tr…

Cited by 0Source ↗Scholar ↗
2023

Training Trajectories of Language Models Across Scales

Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen +2
ACL 2023long

Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In thi…

Cited by 10Source ↗PDF ↗code ↗Scholar ↗
2022

Does Corpus Quality Really Matter for Low-Resource Languages?

Mikel Artetxe, Itziar Aldabe, Rodrigo Agerri, Olatz Perez-de-Viñaspre, Aitor Soroa
EMNLP 2022main

The vast majority of non-English corpora are derived from automatically filtered versions of CommonCrawl. While prior work has identified major issues on the quality of these datasets (Kreutzer et al., 2021), it is not clear how this impacts downstream performance. Taking representation learning in…

Cited by 38Source ↗PDF ↗code ↗Scholar ↗
2022

Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models

Mozes van de Kar, Mengzhou Xia, Danqi Chen, Mikel Artetxe
EMNLP 2022main

Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settin…

Cited by 16Source ↗PDF ↗Scholar ↗
2022

Efficient Large Scale Language Modeling with Mixtures of Experts

Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer +18
EMNLP 2022main

Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language mod…

Cited by 146Source ↗code ↗Scholar ↗
2022

Few-shot Learning with Multilingual Generative Language Models

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig +15
EMNLP 2022main

Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their training data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train…

Cited by 88Source ↗PDF ↗code ↗Scholar ↗
2022

Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel +1
NAACL 2022long

Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping th…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2022

Multilingual Machine Translation with Hyper-Adapters

Christos Baziotis, Mikel Artetxe, James Cross, Shruti Bhosale
EMNLP 2022main

Multilingual machine translation suffers from negative interference across languages. A common solution is to relax parameter sharing with language-specific modules like adapters. However, adapters of related languages are unable to transfer information, and their total number of parameters becomes…

Cited by 24Source ↗PDF ↗code ↗Scholar ↗
2022

On the Role of Bidirectionality in Language Model Pre-Training

Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, Veselin Stoyanov
EMNLP 2022finding

Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. In this work, we focus on bidirectionality as a key factor that differentiates existing approaches,…

Cited by 9Source ↗PDF ↗Scholar ↗
2022

PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining

Machel Reid, Mikel Artetxe
NAACL 2022long

Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora and do not make use of the strong cross-lingual signal contained in parallel data. In this paper, we present PARADISE (PARAllel &Denoising Integration in SEquence-to-sequence mo…

Cited by 23Source ↗PDF ↗code ↗Scholar ↗
2022

PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation

Aitor Ormazabal, Mikel Artetxe, Manex Agirrezabal, Aitor Soroa, Eneko Agirre
EMNLP 2022finding

Formal verse poetry imposes strict constraints on the meter and rhyme scheme of poems. Most prior work on generating this type of poetry uses existing poems for supervision, which are difficult to obtain for most languages and poetic forms. In this work, we propose an unsupervised approach to genera…

Cited by 26Source ↗PDF ↗code ↗Scholar ↗
2022

Principled Paraphrase Generation with Parallel Corpora

Aitor Ormazabal, Mikel Artetxe, Aitor Soroa, Gorka Labaka, Eneko Agirre
ACL 2022long

Round-trip Machine Translation (MT) is a popular choice for paraphrase generation, which leverages readily available parallel corpora for supervision. In this paper, we formalize the implicit similarity function induced by this approach, and show that it is susceptible to non-paraphrase pairs sharin…

Cited by 10Source ↗PDF ↗code ↗Scholar ↗
2022

Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models

Mengzhou Xia, Mikel Artetxe, Jingfei Du, Danqi Chen, Veselin Stoyanov
EMNLP 2022main

Pre-trained masked language models successfully perform few-shot learning by formulating downstream tasks as text infilling. How- ever, as a strong alternative in full-shot settings, discriminative pre-trained models like ELECTRA do not fit into the paradigm. In this work, we adapt prompt-based few-…

Cited by 18Source ↗PDF ↗code ↗Scholar ↗
2022

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi +1
EMNLP 2022main

Large language models (LMs) are able to in-context learn—perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrati…

Cited by 1Source ↗PDF ↗code ↗Scholar ↗
2021

Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring

Aitor Ormazabal, Mikel Artetxe, Aitor Soroa, Gorka Labaka, Eneko Agirre
ACL 2021long

Recent research on cross-lingual word embeddings has been dominated by unsupervised mapping approaches that align monolingual embeddings. Such methods critically rely on those embeddings having a similar structure, but it was recently shown that the separate training in different languages causes de…

Cited by 15Source ↗PDF ↗Scholar ↗