A MoE Multimodal Graph Attention Network Framework for Multimodal Emotion Recognition

Abstract

Multimodal emotion recognition in conversation (ERC) has attracted more attention due to its wide application in multiple fields. Most previous related works focused on directly fusing different modalities, resulting in the excessive introduction of irrelevant multimodal information, which interferes with the model’s prediction. In this paper, we propose a Mixture of Experts Multimodal Graph Attention Network Framework For Multimodal Emotion Recognition (MMGAT-EMO), which combines Multimodal Graph Attention Network (Multimodal GAT) with Mixture of Experts (MoE). According to the semantic relationship of the text modality, it selectively integrates audio and visual features from other nodes into the text modality of the current node and gives different weights to different experts, to handle complex multimodal scenarios. At the same time, we introduce cross-modal Contrastive Loss to shorten the distance between different modalities, for better cross-modal fusion. The MMGAT-EMO method has been evaluated on two widely used multimodal datasets, IEMOCAP and MELD. The results show that MMGAT-EMO is superior to all baseline models and has significantly improved F1-score. We release the code at https://github.com/tdfxlyh/MMGATEMO.