Automatic Document Selection for Efficient Encoder Pretraining
Yukun Feng, Patrick Xia, Benjamin Van Durme, João Sedoc
Abstract
Building pretrained language models is considered expensive and data-intensive, but must we increase dataset size to achieve better performance? We propose an alternative to larger training sets by automatically identifying smaller yet domain-representative subsets. We extend Cynical Data Selection, a statistical sentence scoring method that conditions on a representative target domain corpus. As an example, we treat the OntoNotes corpus as a target domain and pretrain a RoBERTa-like encoder from a cynically selected subset of the Pile. On both perplexity and across several downstream tasks in the target domain, it consistently outperforms random selection with 20x less data, 3x fewer training iterations, and 2x less estimated cloud compute cost, validating the recipe of automatic document selection for LM pretraining.
BibTeX
@inproceedings{feng-etal-2022-automatic,
title = "Automatic Document Selection for Efficient Encoder Pretraining",
author = "Feng, Yukun and
Xia, Patrick and
Van Durme, Benjamin and
Sedoc, Jo{\~a}o",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.647/",
doi = "10.18653/v1/2022.emnlp-main.647",
pages = "9522--9530"
}