Semantic-Aware Prompt Learning for Multimodal Sarcasm Detection

Abstract

Multimodal sarcasm detection aims to identify whether utterances express sarcastic intentions contrary to their literal meaning based on multimodal information. However, existing methods fail to explore the model’s "ability to understand" the semantics expressed by sentences in the image context from semantic diversity perspectives. In this paper, we propose a multi-view semantic awareness method, which concretizes semantics from multiple perspectives to improve the model’s ability to capture different semantic features. Specifically, two learnable prefixes are attached to the text representation respectively to construct semantic representations from both the literal meaning and sarcastic intention perspectives. Then, image-text information is further fused through cross-attention to guide the semantic representation of different perspectives in the image context. Finally, the semantics expressed by prefixes are strengthened through KL divergence, thereby encouraging the model to capture two distinctive semantic features. Experiments on benchmark datasets demonstrate the effectiveness of our method.