DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih, Xilun Chen
Abstract
Large language models (LLMs) have demonstrated strong effectiveness and robustness when fine-tuned as dense retrievers.However, their large parameter size presents significant computational challenges at inference time.While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data.In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers.In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup.Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages.
BibTeX
@inproceedings{ma-etal-2025-drama,
title = "{DRAMA}: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers",
author = "Ma, Xueguang and
Lin, Xi Victoria and
Oguz, Barlas and
Lin, Jimmy and
Yih, Wen-tau and
Chen, Xilun",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.1457/",
doi = "10.18653/v1/2025.acl-long.1457",
pages = "30170--30186",
ISBN = "979-8-89176-251-0"
}