Snapshot-Guided Domain Adaptation for ELECTRA
Daixuan Cheng, Shaohan Huang, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Furu Wei, Denvy Deng, Qi Zhang
Abstract
Discriminative pre-trained language models, such as ELECTRA, have achieved promising performances in a variety of general tasks. However, these generic pre-trained models struggle to capture domain-specific knowledge of domain-related tasks. In this work, we propose a novel domain-adaptation method for ELECTRA, which can dynamically select domain-specific tokens and guide the discriminator to emphasize them, without introducing new training parameters. We show that by re-weighting the losses of domain-specific tokens, ELECTRA can be effectively adapted to different domains. The experimental results in both computer science and biomedical domains show that the proposed method can achieve state-of-the-art results on the domain-related tasks.
BibTeX
@inproceedings{cheng-etal-2022-snapshot,
title = "Snapshot-Guided Domain Adaptation for {ELECTRA}",
author = "Cheng, Daixuan and
Huang, Shaohan and
Liu, Jianfeng and
Zhan, Yuefeng and
Sun, Hao and
Wei, Furu and
Deng, Denvy and
Zhang, Qi",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-emnlp.163/",
doi = "10.18653/v1/2022.findings-emnlp.163",
pages = "2226--2232"
}