SumCSE: Summary as a transformation for Contrastive Learning
Raghuveer Thirukovalluru, Xiaolan Wang, Jun Chen, Shuyang Li, Jie Lei, Rong Jin, Bhuwan Dhingra
Abstract
Sentence embedding models are typically trained using contrastive learning (CL), either using human annotations directly or by repurposing other annotated datasets. In this work, we explore the recently introduced paradigm of generating CL data using generative language models (LM). In CL for computer vision (CV), compositional transformations (series of operations applied over an image. e.g. cropping + color distortion) which modify the input/image to retain minimal information were shown to be very effective. We show that composition of a ‘Summary’ transformation with diverse paraphrasing/contradicting transformations accomplishes the same and works very well in CL for sentence embeddings. Our final generated dataset (using Vicuna-13B) significantly outperforms the previous best unsupervised method (using ChatGPT) by 1.8 points, and SimCSE, a strong supervised baseline by 0.3 points on the semantic text similarity (STS) benchmark.
BibTeX
@inproceedings{thirukovalluru-etal-2024-sumcse,
title = "{S}um{CSE}: Summary as a transformation for Contrastive Learning",
author = "Thirukovalluru, Raghuveer and
Wang, Xiaolan and
Chen, Jun and
Li, Shuyang and
Lei, Jie and
Jin, Rong and
Dhingra, Bhuwan",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-naacl.227/",
doi = "10.18653/v1/2024.findings-naacl.227",
pages = "3577--3588"
}