The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics
Ricardo Rei, Nuno M. Guerreiro, Marcos Treviso, Luisa Coheur, Alon Lavie, André Martins
Abstract
Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU. Yet, neural metrics are, to a great extent, “black boxes” returning a single sentence-level score without transparency about the decision-making process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at: https://github.com/Unbabel/COMET/tree/explainable-metrics
BibTeX
@inproceedings{rei-etal-2023-inside,
title = "The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics",
author = "Rei, Ricardo and
Guerreiro, Nuno M. and
Treviso, Marcos and
Coheur, Luisa and
Lavie, Alon and
Martins, Andr{\'e}",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-short.94/",
doi = "10.18653/v1/2023.acl-short.94",
pages = "1089--1105"
}