Momentum Contrastive Pre-training for Question Answering
Minda Hu, Muzhi Li, Yasheng Wang, Irwin King
Abstract
Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In order to address this problem, we propose a novel Momentum Contrastive pRe-training fOr queStion anSwering (MCROSS) method for extractive QA. Specifically, MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Hence, the pre-trained models can better transfer the knowledge learned in cloze-like samples to answering natural questions. Experimental results on three benchmarking QA datasets show that our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.
BibTeX
@inproceedings{hu-etal-2022-momentum,
title = "Momentum Contrastive Pre-training for Question Answering",
author = "Hu, Minda and
Li, Muzhi and
Wang, Yasheng and
King, Irwin",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.291/",
doi = "10.18653/v1/2022.emnlp-main.291",
pages = "4324--4330"
}