Multimodal Dialogue Emotion Recognition Based on Label Optimization and Coarse-Grained Assisted Fine-Grained

Abstract

Multimodal dialogue emotion recognition integrates data from multiple modalities to accurately identify emotional states in conversations. However, differences in expression and information density across modalities complicate the fusion of features. Traditional methods may introduce redundant information from other utterances, reducing the accuracy of emotion recognition. Existing one-hot labels often fail to capture the full range of emotional expressions, leading to biased results. To address these issues, we propose a model that fuses different modalities within the same utterance to avoid redundancy. It employs a progressive classification process, refining emotion recognition from coarse to fine granularity. Additionally, we use emotion polarity probabilities as weights for fine-grained classification and introduce a multimodal information-rich label that considers both the data and their interactions. Experiments on IEMOCAP and MELD datasets demonstrate the model’s effectiveness, significantly improving dialog emotion recognition accuracy. Our code is available at https://anonymous.4open.science/r/LOCG-188E.