NAACL 2021long22 citations
Assessing Reference-Free Peer Evaluation for Machine Translation
Sweta Agrawal, George Foster, Markus Freitag, Colin Cherry
Abstract
Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains. It has been recently shown that the probabilities given by a large, multilingual model can achieve state of the art results when used as a reference-free metric. We experiment with various modifications to this model, and demonstrate that by scaling it up we can match the performance of BLEU. We analyze various potential weaknesses of the approach, and find that it is surprisingly robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities.
BibTeX
@inproceedings{agrawal-etal-2021-assessing,
title = "Assessing Reference-Free Peer Evaluation for Machine Translation",
author = "Agrawal, Sweta and
Foster, George and
Freitag, Markus and
Cherry, Colin",
editor = "Toutanova, Kristina and
Rumshisky, Anna and
Zettlemoyer, Luke and
Hakkani-Tur, Dilek and
Beltagy, Iz and
Bethard, Steven and
Cotterell, Ryan and
Chakraborty, Tanmoy and
Zhou, Yichao",
booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jun,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.naacl-main.91/",
doi = "10.18653/v1/2021.naacl-main.91",
pages = "1158--1171"
}