MSACC: A Unified Multimodal Sentiment Analysis Framework for High Interpretability and Zero-shot Performance

Abstract

Compared to large language models, traditional multimodal sentiment analysis frameworks are constrained by their classification heads, resulting in poor performance on zero-shot tasks. Moreover, due to limitations in visual encoders and multimodal fusion modules, most existing frameworks can only process a small number of images, leading to a loss of visual information. In light of these issues, this paper proposes a new framework, MSACC. This framework enhances the model’s zero-shot performance by adopting a contrastive classification method and reduces the loss of visual information through visual relation extraction and three-dimensional sentiment analysis. We conducted extensive experiments on the Yelp dataset. The experimental results show that MSACC outperforms models of the same category in zero-shot MSA tasks, achieving a 48% performance improvement. Furthermore, compared to the large language model ChatGLM2-6B, MSACC still achieved a 7% performance increase while saving 90% of the model size. In addition, in supervised tasks, MSACC also achieved a 3.27% performance improvement compared to the baseline model.