MSA-ITEI: A Novel Method for Multimodal Analysis of Social Media Stickers

Abstract

In social media, stickers are commonly used alongside text to convey sentiment and intent, yet their abstract nature and embedded text create challenges for multimodal analysis. Current research is limited by these characteristics and the lack of datasets. To address this gap, we introduce MSA-ITEI: Multimodal Sticker Analysis through Image, Text, and underlying Emotions and Intentions. This method effectively generates accurate text descriptions of stickers and handles multimodal tasks in a textual format. Experiments on two multimodal sticker datasets demonstrate that MSA-ITEI outperforms other leading multimodal models and large language models, in tasks such as sticker sentiment analysis, multimodal sentiment analysis, and intent recognition. Our code will be made publicly available.