AIConfPaper
← Search

Eleftheria Briakou

12 accepted papers

2025

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation

Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag
ICML 2025poster

Data contamination—the accidental consumption of evaluation examples within the pre-training data—can undermine the validity of evaluation benchmarks. In this paper, we present a rigorous analysis of the effects of contamination on language models at 1B and 8B scales on the machine translation task.…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?

Senyu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou +4
EMNLP 2025

Evaluating machine translation (MT) quality for under-resourced African languages remains a significant challenge, as existing metrics often suffer from limited language coverage and poor performance in low-resource settings. While recent efforts, such as AfriCOMET, have addressed some of the issues

Cited by 0Source ↗PDF ↗Scholar ↗
2025

WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects

Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska +11
ACL 2025finding

As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual performance, including on tasks like machine translation (MT). In this work, we extend the WMT24 dataset to cover 55 lang…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou +52
NAACL 2024long

Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation wi…

Cited by 6Source ↗PDF ↗code ↗Scholar ↗
2023

Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences

Eleftheria Briakou, Navita Goyal, Marine Carpuat
EMNLP 2023long main

Explainable NLP techniques primarily explain by answering "Which tokens in the input are responsible for this prediction?". We argue that for NLP models that make predictions by comparing two input texts, it is more useful to explain by answering "What differences between the two inputs explain this…

Cited by 0Source ↗code ↗Scholar ↗
2023

Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM’s Translation Capability

Eleftheria Briakou, Colin Cherry, George Foster
ACL 2023long

Large, multilingual language models exhibit surprisingly good zero- or few-shot machine translation capabilities, despite having never seen the intentionally-included translation examples provided to typical neural translation systems. We investigate the role of incidental bilingualism—the unintenti…

Cited by 59Source ↗PDF ↗Scholar ↗
2023

What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems

Navita Goyal, Eleftheria Briakou, Amanda Stephanie Liu, Connor Baumler, Claire Bonial, Jeffrey Micher +3
EMNLP 2023long main

NLP systems have shown impressive performance at answering questions by retrieving relevant context. However, with the increasingly large models, it is impossible and often undesirable to constrain models' knowledge or reasoning to only the retrieved context. This leads to a mismatch between the inf…

Cited by 0Source ↗Scholar ↗
2022

BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation

Eleftheria Briakou, Sida Wang, Luke Zettlemoyer, Marjan Ghazvininejad
NAACL 2022findings

Mined bitexts can contain imperfect translations that yield unreliable training signals for Neural Machine Translation (NMT). While filtering such pairs out is known to improve final model quality, we argue that it is suboptimal in low-resource conditions where even mined data can be limited. In our…

Cited by 7Source ↗PDF ↗code ↗Scholar ↗
2021

Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation

Eleftheria Briakou, Marine Carpuat
ACL 2021long

While it has been shown that Neural Machine Translation (NMT) is highly sensitive to noisy parallel training samples, prior work treats all types of mismatches between source and target as noise. As a result, it remains unclear how samples that are mostly equivalent but contain a small number of sem…

Cited by 16Source ↗PDF ↗code ↗Scholar ↗
2021

Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

Eleftheria Briakou, Sweta Agrawal, Joel Tetreault, Marine Carpuat
EMNLP 2021main

While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading automatic metrics on the oft-researched task of formality style transfer. Unlike previous evaluations, which focus so…

Cited by 32Source ↗PDF ↗code ↗Scholar ↗
2021

Olá, Bonjour, Salve! XFORMAL: A Benchmark for Multilingual Formality Style Transfer

Eleftheria Briakou, Di Lu, Ke Zhang, Joel Tetreault
NAACL 2021long

We take the first step towards multilingual style transfer by creating and releasing XFORMAL, a benchmark of multiple formal reformulations of informal text in Brazilian Portuguese, French, and Italian. Results on XFORMAL suggest that state-of-the-art style transfer approaches perform close to simpl…

Cited by 53Source ↗PDF ↗code ↗Scholar ↗