What Language Do Non-English-Centric Large Language Models Think in?
Chengzhi Zhong, Qianying Liu, Fei Cheng, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, Sadao Kurohashi
Abstract
In this study, we investigate whether non-English-centric large language models, ‘think’ in their specialized language. Specifically, we analyze how intermediate layer representations, when projected into the vocabulary space, favor certain languages during generation—termed as latent languages. We categorize non-English-centric models into two groups: CPMs, which are English-centric models with continued pre-training on its specialized language, and BLMs, which are pre-trained on a balanced mix of multiple languages from scratch. Our findings reveal that while English-centric models rely exclusively on English as their latent language, non-English-centric models activate multiple latent languages, dynamically selecting the most similar one based on both the source and target languages. This also influences responses to culture difference questions, reducing English-centric biases in non-English models. This study deepens our understanding of language representation in non-English-centric LLMs, shedding light on the intricate dynamics of multilingual processing at the representational level.
BibTeX
@inproceedings{zhong-etal-2025-language,
title = "What Language Do Non-{E}nglish-Centric Large Language Models Think in?",
author = "Zhong, Chengzhi and
Liu, Qianying and
Cheng, Fei and
Jiang, Junfeng and
Wan, Zhen and
Chu, Chenhui and
Murawaki, Yugo and
Kurohashi, Sadao",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.1350/",
doi = "10.18653/v1/2025.findings-acl.1350",
pages = "26333--26346",
ISBN = "979-8-89176-256-5"
}