Representational Isomorphism and Alignment of Multilingual Large Language Models
Di Wu, Yibin Lei, Andrew Yates, Christof Monz
Abstract
In this paper, we investigate the capability of Large Language Models (LLMs) to represent texts in multilingual contexts. Our findings show that sentence representations derived from LLMs exhibit a high degree of isomorphism across languages.This existing isomorphism can facilitate representational alignments in zero-shot and few-shot settings.Specifically, by applying a contrastive objective at the representation level with only a small number of translation pairs (e.g., 100), we substantially improve models’ performance on Semantic Textual Similarity (STS) tasks across languages. This representation-level approach proves to be more efficient and effective for semantic alignment than continued pretraining or instruction tuning. Interestingly, we also observe substantial STS improvements within individual languages, even without a monolingual objective specifically designed for this purpose.
BibTeX
@inproceedings{wu-etal-2024-representational,
title = "Representational Isomorphism and Alignment of Multilingual Large Language Models",
author = "Wu, Di and
Lei, Yibin and
Yates, Andrew and
Monz, Christof",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.823/",
doi = "10.18653/v1/2024.findings-emnlp.823",
pages = "14074--14085"
}