Prompt Fusion and Aspect-Oriented Filtration for Aspect-Based Multimodal Sentiment Analysis

Abstract

Currently, aspect-based multimodal sentiment analysis (ABMSA) remains a highly hot research field, aiming to leverage various modalities such as images and text to determine the sentiment orientation of viewpoint entities. Although deep learning methods have made significant progress in this field, some challenges still exist: Image information unrelated to the viewpoint entity can introduce noise, insufficient interaction during modal fusion. To solve these problems, we propose a novel ABMSA model with prompt fusion and noise filtration. Specifically, we introduced an Image Noise Filtering (INF) module that can filter the extracted image features to reduce the noise introduced by the irrelevant between viewpoint entity and the image. Additionally, we implemented Multimodal Prompt Fusion (MPF) module to enhance the interaction between modalities, addressing the issue of minimal effective features related to viewpoint entity during the fusion stage of text and image features. Experimental results show that the model outperforms the baseline model on two public datasets, Twitter-2015 and Twitter-2017, demonstrating that our proposed approach effectively enhances the accuracy of aspect-based multimodal sentiment analysis tasks.