SwiLTra-Bench: The Swiss Legal Translation Benchmark
Joel Niklaus, Jakob Merane, Luka Nenadic, Sina Ahmadi, Yingqiang Gao, Cyrill A. H. Chevalley, Claude Humbel, Christophe Gösken
Abstract
In Switzerland legal translation is uniquely important due to the country’s four official languages and requirements for multilingual legal documentation. However, this process traditionally relies on professionals who must be both legal experts and skilled translators—creating bottlenecks and impacting effective access to justice. To address this challenge, we introduce SwiLTra-Bench, a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems. Our systematic evaluation reveals that frontier models achieve superior translation performance across all document types, while specialized translation systems excel specifically in laws but under-perform in headnotes. Through rigorous testing and human expert validation, we demonstrate that while fine-tuning open SLMs significantly improves their translation quality, they still lag behind the best zero-shot prompted frontier models such as Claude-3.5-Sonnet. Additionally, we present SwiLTra-Judge, a specialized LLM evaluation system that aligns best with human expert assessments.
BibTeX
@inproceedings{niklaus-etal-2025-swiltra,
title = "{S}wi{LT}ra-Bench: The {S}wiss Legal Translation Benchmark",
author = {Niklaus, Joel and
Merane, Jakob and
Nenadic, Luka and
Ahmadi, Sina and
Gao, Yingqiang and
Chevalley, Cyrill A. H. and
Humbel, Claude and
G{\"o}sken, Christophe and
Tanzi, Lorenzo and
L{\"u}thi, Thomas and
Palombo, Stefan and
Poff, Spencer and
Yang, Boling and
Wu, Nan and
Guillod, Matthew and
Mami{\'e}, Robin and
Brunner, Daniel and
Pereyra, Julio and
Grupen, Niko},
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.725/",
doi = "10.18653/v1/2025.acl-long.725",
pages = "14894--14916",
ISBN = "979-8-89176-251-0"
}