Intra-modal Relation and Emotional Incongruity Learning using Graph Attention Networks for Multimodal Sarcasm Detection
Devraj Raghuvanshi, Xiyuan Gao, Zhu Li, Shubhi Bansal, Matt Coler, Nagendra Kumar, Shekhar Nayak
Abstract
Sarcasm detection poses unique challenges due to the complex nature of sarcastic expressions often embedded across multiple modalities. Current methods frequently fall short in capturing the incongruent emotional cues that are essential for identifying sarcasm in multimodal contexts. In this paper, we present a novel method to capture the pair-wise emotional incongruities between modalities through a cross-modal Contrastive Attention Mechanism (CAM), leveraging advanced data augmentation techniques to enhance data diversity and Supervised Contrastive Learning (SCL) to obtain discriminative embeddings. Additionally, we employ Graph Attention Networks (GATs) to construct modality-specific graphs, capturing intra-modal dependencies. Experiments conducted on the MUStARD++ dataset demonstrate the efficacy of our approach, achieving a macro F1 score of 74.96%, which outperforms state-of-the-art methods.
BibTeX
@inproceedings{icassp2025_intramodalrelati,
title = {Intra-modal Relation and Emotional Incongruity Learning using Graph Attention Networks for Multimodal Sarcasm Detection},
author = {Devraj Raghuvanshi and Xiyuan Gao and Zhu Li and Shubhi Bansal and Matt Coler and Nagendra Kumar and Shekhar Nayak},
booktitle = {ICASSP 2025},
year = {2025}
}