A Fine-Grained Tri-Modal Interaction Model for Multimodal Sentiment Analysis

Abstract

The methods based on multimodal representation learning enhance discriminable sentiment expression for multimodal sentiment analysis(MSA). The modal invariant and specific features serve different purposes in sentiment learning and the diversity of inter-sample and inter-category relationships takes less consideration in previous advances. In this paper, we propose a fine-grained tri-modal interaction model for MSA to refine and enhance the overall affective state at the uni/multi-modal level and label level. Concretely, we simultaneously focus on the contributions of both unimodal and multimodal views to improve holistic affective knowledge. The similarity measurement function and self-supervised learning are introduced to reduce the inherent modality gap and refine the invariant representations. We provide two specific constraints to strengthen the specific uniqueness and discriminative capacity. Moreover, we develop a multi-granularity fusion module to fully integrate two views in a coarse-to-fine form. Experimental results on two datasets show that our method fares better than the state-of-the-art model.