MUSTS: MUltilingual Semantic Textual Similarity Benchmark
Tharindu Ranasinghe, Hansi Hettiarachchi, Constantin Orasan, Ruslan Mitkov
Abstract
Predicting semantic textual similarity (STS) is a complex and ongoing challenge in natural language processing (NLP). Over the years, researchers have developed a variety of supervised and unsupervised approaches to calculate STS automatically. Additionally, various benchmarks, which include STS datasets, have been established to consistently evaluate and compare these STS methods. However, they largely focus on high-resource languages, mixed with datasets annotated focusing on relatedness instead of similarity and containing automatically translated instances. Therefore, no dedicated benchmark for multilingual STS exists. To solve this gap, we introduce the Multilingual Semantic Textual Similarity Benchmark (MUSTS), which spans 13 languages, including low-resource languages. By evaluating more than 25 models on MUSTS, we establish the most comprehensive benchmark of multilingual STS methods. Our findings confirm that STS remains a challenging task, particularly for low-resource languages.
BibTeX
@inproceedings{ranasinghe-etal-2025-musts,
title = "{MUSTS}: {MU}ltilingual Semantic Textual Similarity Benchmark",
author = "Ranasinghe, Tharindu and
Hettiarachchi, Hansi and
Orasan, Constantin and
Mitkov, Ruslan",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-short.27/",
doi = "10.18653/v1/2025.acl-short.27",
pages = "331--353",
ISBN = "979-8-89176-252-7"
}