A Fair Comparison without Translationese: English vs. Target-language Instructions for Multilingual LLMs
Taisei Enomoto, Hwichan Kim, Zhousi Chen, Mamoru Komachi
Abstract
Most large language models are multilingual instruction executors. Prior studies suggested that English instructions are more effective than target-language instructions even for non-English tasks; however, these studies often use datasets and instructions translated from English, which introduce biases known as translationese, hindering an unbiased comparison. To address this issue, we conduct a fair comparison between English and target-language instructions by eliminating translationese effects. Contrary to previous studies, our experiments across several tasks reveal that the advantage of adopting English instructions is not overwhelming. Additionally, we report on the features of generated texts and the instruction-following abilities when using respective instructions.
BibTeX
@inproceedings{enomoto-etal-2025-fair,
title = "A Fair Comparison without Translationese: {E}nglish vs. Target-language Instructions for Multilingual {LLM}s",
author = "Enomoto, Taisei and
Kim, Hwichan and
Chen, Zhousi and
Komachi, Mamoru",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.naacl-short.55/",
pages = "649--670",
ISBN = "979-8-89176-190-2"
}