Domain-matched Pre-training Tasks for Dense Retrieval
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel
Abstract
Pre-training on larger datasets with ever increasing model size isnow a proven recipe for increased performance across almost all NLP tasks.A notable exception is information retrieval, where additional pre-traininghas so far failed to produce convincing results. We show that, with theright pre-training setup, this barrier can be overcome. We demonstrate thisby pre-training large bi-encoder models on 1) a recently released set of 65 millionsynthetically generated questions, and 2) 200 million post-comment pairs from a preexisting dataset of Reddit conversations made available by pushshift.io. We evaluate on a set of information retrieval and dialogue retrieval benchmarks, showing substantial improvements over supervised baselines.
BibTeX
@inproceedings{oguz-etal-2022-domain,
title = "Domain-matched Pre-training Tasks for Dense Retrieval",
author = "Oguz, Barlas and
Lakhotia, Kushal and
Gupta, Anchit and
Lewis, Patrick and
Karpukhin, Vladimir and
Piktus, Aleksandra and
Chen, Xilun and
Riedel, Sebastian and
Yih, Scott and
Gupta, Sonal and
Mehdad, Yashar",
editor = "Carpuat, Marine and
de Marneffe, Marie-Catherine and
Meza Ruiz, Ivan Vladimir",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-naacl.114/",
doi = "10.18653/v1/2022.findings-naacl.114",
pages = "1524--1534"
}