AIConfPaper
← Search

Pierre Andrews

7 accepted papers

2026

Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments

Romain Froger, Amine Benhalloum, Andrey Rusakov, Dheeraj Mekala, Emilien Garreau, Gerard Moreno-Torres Bertran +18
ICLR 2026oral

We introduce **Gaia2**, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal cons…

Cited by 0Source ↗Scholar ↗
2025

2M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Dataset Download PDF

Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz, Joe Chuang +4
ACL 2025finding

We introduce the first highly multilingual speech and American Sign Language (ASL) comprehension dataset by extending BELEBELE. Our dataset covers 91 spoken languages at the intersection of BELEBELE and FLEURS, and one sign language (ASL). As a by-product we also extend the Automatic Speech Recognit…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

BOUQuET : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation

Pierre Andrews, Mikel Artetxe, Mariano Coria Meglioli, Marta R. Costa-juss{\`a}, Joe Chuang, David Dale +13
EMNLP 2025

BOUQuET is a multi-way, multicentric and multi-register/domain dataset and benchmark, and a broader collaborative initiative. This dataset is handcrafted in 8 non-English languages (i.e. Egyptian Arabic and Modern Standard Arabic, French, German, Hindi, Indonesian, Mandarin Chinese, Russian, and Spa

Cited by 0Source ↗PDF ↗Scholar ↗
2025

LCFO: Long Context and Long Form Output Dataset and Benchmarking

Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen, Joe Chuang, David Dale +7
ACL 2025finding

This paper presents the Long Context and Form Output (LCFO) benchmark, a novel evaluation framework for assessing gradual summarization and summary expansion capabilities across diverse domains. LCFO consists of long input documents (5k words average length), each of which comes with three summaries…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector

Marta Costa-jussà, Mariano Meglioli, Pierre Andrews, David Dale, Prangthip Hansanti, Elahe Kalbassi +3
ACL 2024findings

Research in toxicity detection in natural language processing for the speech modality (audio-based) is quite limited, particularly for languages other than English. To address these limitations and lay the groundwork for truly multilingual audio-based toxicity detection, we introduce MuTox, the firs…

Cited by 10Source ↗PDF ↗code ↗Scholar ↗
2023

BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric

Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk +1
ACL 2023long

End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propos…

Cited by 21Source ↗PDF ↗code ↗Scholar ↗
2023

Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale

Marta R. Costa-jussà, Pierre Andrews, Eric Michael Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi +3
EMNLP 2023long main

We introduce a multilingual extension of the HolisticBias dataset, the largest English template-based taxonomy of textual people references: Multilingual HolisticBias. This extension consists of 20,459 sentences in 50 languages distributed across 13 demographic axes. Source sentences are built from…

Cited by 0Source ↗Scholar ↗