Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models
Ahmed Elshabrawy, Thanh-Nhi Nguyen, Yeeun Kang, Lihan Feng, Annant Jain, Faadil Abdullah Shaikh, Jonibek Mansurov, Mohamed Fazli Mohamed Imam
Abstract
Large Language Models (LLMs) excel in zero-shot and few-shot tasks, but achieving similar performance with encoder-only models like BERT and RoBERTa has been challenging due to their architecture. However, encoders offer advantages such as lower computational and memory costs. Recent work adapts them for zero-shot generalization using Statement Tuning, which reformulates tasks into finite templates. We extend this approach to multilingual NLP, exploring whether encoders can achieve zero-shot cross-lingual generalization and serve as efficient alternatives to memory-intensive LLMs for low-resource languages. Our results show that state-of-the-art encoder models generalize well across languages, rivaling multilingual LLMs while being more efficient. We also analyze multilingual Statement Tuning dataset design, efficiency gains, and language-specific generalization, contributing to more inclusive and resource-efficient NLP models. We release our code and models.
BibTeX
@inproceedings{elshabrawy-etal-2025-statement,
title = "Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models",
author = "Elshabrawy, Ahmed and
Nguyen, Thanh-Nhi and
Kang, Yeeun and
Feng, Lihan and
Jain, Annant and
Shaikh, Faadil Abdullah and
Mansurov, Jonibek and
Imam, Mohamed Fazli Mohamed and
Ortiz-Barajas, Jesus-German and
Chevi, Rendi and
Aji, Alham Fikri",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.835/",
doi = "10.18653/v1/2025.findings-acl.835",
pages = "16226--16248",
ISBN = "979-8-89176-256-5"
}