SeaLLMs - Large Language Models for Southeast Asia
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li
Abstract
Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon popular English-centric models through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.
BibTeX
@inproceedings{nguyen-etal-2024-seallms,
title = "{S}ea{LLM}s - Large Language Models for {S}outheast {A}sia",
author = "Nguyen, Xuan-Phi and
Zhang, Wenxuan and
Li, Xin and
Aljunied, Mahani and
Hu, Zhiqiang and
Shen, Chenhui and
Chia, Yew Ken and
Li, Xingxuan and
Wang, Jianyu and
Tan, Qingyu and
Cheng, Liying and
Chen, Guanzheng and
Deng, Yue and
Yang, Sen and
Liu, Chaoqun and
Zhang, Hang and
Bing, Lidong",
editor = "Cao, Yixin and
Feng, Yang and
Xiong, Deyi",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-demos.28/",
doi = "10.18653/v1/2024.acl-demos.28",
pages = "294--304"
}