Pruning Multilingual Large Language Models for Multilingual Inference

Hwichan Kim; Jun Suzuki; Tosho Hirasawa; Mamoru Komachi

EMNLP 2024finding0 citations

Pruning Multilingual Large Language Models for Multilingual Inference

Hwichan Kim, Jun Suzuki, Tosho Hirasawa, Mamoru Komachi

Abstract

Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. This study introduces a promising direction for enhancing non-English performance through a specialized pruning approach. Specifically, we prune MLLMs using bilingual sentence pairs from English and other languages and empirically demonstrate that this pruning strategy can enhance the MLLMs’ performance in non-English language.

BibTeX

@inproceedings{kim-etal-2024-pruning,
    title = "Pruning Multilingual Large Language Models for Multilingual Inference",
    author = "Kim, Hwichan  and
      Suzuki, Jun  and
      Hirasawa, Tosho  and
      Komachi, Mamoru",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.580/",
    doi = "10.18653/v1/2024.findings-emnlp.580",
    pages = "9921--9942"
}