Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion
Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Mosen Alharthi, Bang An
Abstract
This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or GPT-3.5, due to a predominant focus on mainstream languages (e.g., English and Chinese). One practical objective for Arabic LLMs is to utilize Arabic-specific vocabulary in the tokenizer to accelerate decoding. However, using a different vocabulary often leads to degradation of the model’s learned knowledge, since many words become out-of-vocabulary (OOV) at the beginning of training. Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion, which is implemented by a modified BPE algorithm that progressively extends the Arabic subwords in its dynamic vocabulary during training, thereby balancing the OOV ratio at every stage. The ablation study demonstrated the effectiveness of Progressive Vocabulary Expansion.Moreover, AraLLaMA achieves decent performance comparable to the best Arabic LLMs across a variety of Arabic benchmarks. Our model weights are available at: https://github.com/FreedomIntelligence/AraLLaMa.
BibTeX
@inproceedings{zhu-etal-2025-second,
title = "Second Language ({A}rabic) Acquisition of {LLM}s via Progressive Vocabulary Expansion",
author = "Zhu, Jianqing and
Huang, Huang and
Lin, Zhihang and
Liang, Juhao and
Tang, Zhengyang and
Almubarak, Khalid and
Alharthi, Mosen and
An, Bang and
He, Juncai and
Wu, Xiangbo and
Yu, Fei and
Chen, Junying and
Zhuoheng, Ma and
Du, Yuhao and
Zhang, He and
Alshahrani, Saied and
Alghamdi, Emad A. and
Zhang, Lian and
Sun, Ruoyu and
Li, Haizhou and
Wang, Benyou and
Xu, Jinchao",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.100/",
doi = "10.18653/v1/2025.acl-long.100",
pages = "2025--2042",
ISBN = "979-8-89176-251-0"
}