ACL 2024short11 citations
Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains
Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, Brian Thompson
Abstract
We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to both metrics that rely on the surface form and pre-trained metrics that are not fine-tuned on MT quality judgments.
BibTeX
@inproceedings{zouhar-etal-2024-fine,
title = "Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains",
author = "Zouhar, Vil{\'e}m and
Ding, Shuoyang and
Currey, Anna and
Badeka, Tatyana and
Wang, Jenyuan and
Thompson, Brian",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-short.45/",
doi = "10.18653/v1/2024.acl-short.45",
pages = "488--500"
}