Self-Guided Contrastive Learning for BERT Sentence Representations
Taeuk Kim, Kang Min Yoo, Sang-goo Lee
Abstract
Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the quality of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embeddings to function as sentence vectors. Moreover, we redesign the contrastive learning objective (NT-Xent) and apply it to sentence representation learning. We demonstrate with extensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is efficient at inference and robust to domain shifts.
BibTeX
@inproceedings{kim-etal-2021-self,
title = "Self-Guided Contrastive Learning for {BERT} Sentence Representations",
author = "Kim, Taeuk and
Yoo, Kang Min and
Lee, Sang-goo",
editor = "Zong, Chengqing and
Xia, Fei and
Li, Wenjie and
Navigli, Roberto",
booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
month = aug,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.acl-long.197/",
doi = "10.18653/v1/2021.acl-long.197",
pages = "2528--2540"
}