Enhancing Speech Emotion Recognition with Speech Dynamic Modeling and Multi-Modal Knowledge Distillation

Abstract

Complementary semantic information from the text modality, obtained through runtime transcription, plays a crucial role in Speech Emotion Recognition (SER). However, it introduces additional computational overhead and potential errors. To address these issues, we propose the SDMMKD framework, which directly leverages multimodal knowledge without runtime transcription. Specifically, SDMMKD distills emotion knowledge at both the feature and logit levels from a pre-trained multimodal teacher during training. During inference, SDMMKD relies solely on speech signals to perform unimodal SER. Additionally, we utilize a Mamba block to enhance dynamic temporal features. Experimental results on the widely used IEMOCAP dataset demonstrate that our proposed SDMMKD framework outperforms state-of-the-art methods, achieving a WAR of 75.89% and a UAR of 77.37%.