A Family of Pretrained Transformer Language Models for Russian
Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko
Abstract
Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we aim to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.
BibTeX
@inproceedings{zmitrovich-etal-2024-family,
title = "A Family of Pretrained Transformer Language Models for {R}ussian",
author = "Zmitrovich, Dmitry and
Abramov, Aleksandr and
Kalmykov, Andrey and
Kadulin, Vitaly and
Tikhonova, Maria and
Taktasheva, Ekaterina and
Astafurov, Danil and
Baushenko, Mark and
Snegirev, Artem and
Shavrina, Tatiana and
Markov, Sergei S. and
Mikhailov, Vladislav and
Fenogenova, Alena",
editor = "Calzolari, Nicoletta and
Kan, Min-Yen and
Hoste, Veronique and
Lenci, Alessandro and
Sakti, Sakriani and
Xue, Nianwen",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
month = may,
year = "2024",
address = "Torino, Italia",
publisher = "ELRA and ICCL",
url = "https://aclanthology.org/2024.lrec-main.45/",
pages = "507--524"
}