Improving the Modality Representation with multi-view Contrastive Learning for Multimodal Sentiment Analysis

Abstract

Modality representation learning is an important problem for multimodal sentiment analysis (MSA), since the highly distinguishable representations can contribute to improving the analysis effect. Previous works of MSA have usually focused on internal fusion strategies for different modalities within one sample, and the external usage of cross reference relations among different samples was given less attention. Recently, the rise of contrastive learning provides powerful clues for us to learn modal representation with stronger discriminative ability. In this study, we explore the approach of representations improvement and devise a three-stages framework with multi-view contrastive learning to refine representations for the specific objectives. Firstly, for each modality, we employ the supervised contrastive learning to pull samples within the same class together while the other samples are pushed apart. Then, a self-supervised contrastive learning is designed for the distilled cross-modal representations after a novel Transformer-based interaction module. At last, we leverage again the supervised contrastive learning to enhance the fused multimodal representation. We conduct extensive experiments on three open datasets, and results show the advance of our model.