AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages
Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He
Abstract
Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).
BibTeX
@inproceedings{wang-etal-2024-afrimte,
title = "{A}fri{MTE} and {A}fri{COMET}: Enhancing {COMET} to Embrace Under-resourced {A}frican Languages",
author = "Wang, Jiayi and
Adelani, David Ifeoluwa and
Agrawal, Sweta and
Masiak, Marek and
Rei, Ricardo and
Briakou, Eleftheria and
Carpuat, Marine and
He, Xuanli and
Bourhim, Sofia and
Bukula, Andiswa and
Mohamed, Muhidin and
Olatoye, Temitayo and
Adewumi, Tosin and
Mokayed, Hamam and
Mwase, Christine and
Kimotho, Wangui and
Yuehgoh, Foutse and
Aremu, Anuoluwapo and
Ojo, Jessica and
Muhammad, Shamsuddeen Hassan and
Osei, Salomey and
Omotayo, Abdul-Hakeem and
Chukwuneke, Chiamaka and
Ogayo, Perez and
Hourrane, Oumaima and
El Anigri, Salma and
Ndolela, Lolwethu and
Mangwana, Thabiso and
Mohamed, Shafie Abdi and
Ayinde, Hassan and
Awoyomi, Oluwabusayo Olufunke and
Alkhaled, Lama and
Al-azzawi, Sana and
Etori, Naome A. and
Ochieng, Millicent and
Siro, Clemencia and
Kiragu, Njoroge and
Muchiri, Eric and
Kimotho, Wangari and
Wamba Momo, Lyse Naomi and
Abolade, Daud and
Ajao, Simbiat and
Shode, Iyanuoluwa and
Macharm, Ricky and
Iro, Ruqayya Nasir and
Abdullahi, Saheed S. and
Moore, Stephen E. and
Opoku, Bernard and
Akinjobi, Zainab and
Afolabi, Abeeb and
Obiefuna, Nnaemeka and
Ogbu, Onyekachi Raphael and
Ochieng{'}, Sam and
Otiende, Verrah Akinyi and
Mbonu, Chinedu Emmanuel and
Toadoum Sari, Sakayo and
Lu, Yao and
Stenetorp, Pontus",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.naacl-long.334/",
doi = "10.18653/v1/2024.naacl-long.334",
pages = "5997--6023"
}