TEMA: Token Embeddings Mapping for Enriching Low-Resource Language Models
Rodolfo Zevallos, Núria Bel, Mireia Farrús
Abstract
The objective of the research we present is to remedy the problem of the low quality of language models for low-resource languages. We introduce an algorithm, the Token Embedding Mapping Algorithm (TEMA), that maps the token embeddings of a richly pre-trained model L1 to a poorly trained model L2, thus creating a richer L2’ model. Our experiments show that the L2’ model reduces perplexity with respect to the original monolingual model L2, and that for downstream tasks, including SuperGLUE, the results are state-of-the-art or better for the most semantic tasks. The models obtained with TEMA are also competitive or better than multilingual or extended models proposed as solutions for mitigating the low-resource language problems.
BibTeX
@inproceedings{zevallos-etal-2024-tema,
title = "{TEMA}: Token Embeddings Mapping for Enriching Low-Resource Language Models",
author = "Zevallos, Rodolfo and
Bel, N{\'u}ria and
Farr{\'u}s, Mireia",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-main.638/",
doi = "10.18653/v1/2024.emnlp-main.638",
pages = "11423--11435"
}