Exploiting Wavelet Scattering Transform & Squeeze-Excitation Blocks with Cross-Modal Attention for Multi-modal Emotion Recognition

Abstract

Multi-modal emotion recognition (MER) is crucial for improving human-computer interaction. Convolutional neural networks (CNNs) are the mainstream for MER tasks, but they require large databases, extensive memory, and significant energy, limiting their practical use. This paper proposes a novel MER system that leverages wavelet scattering transform (WST) to address these challenges, achieving improved performance with lower computational consumption. Moreover, the system benefits from the noise robustness provided by WST. By integrating WST as a non-trainable initial layer in a CNN model and employing an encoder module, our system effectively captures time-frequency, local and high-level features from both speech and video. We enhance feature integration and representation with cross-modal attention (CMA) and a squeeze-and-excitation (SE) block. The results demonstrate that our system performs consistently across varying noise levels and duration thresholds. Ablation studies reveal that the combination of MFCC, Mel spectrogram, and raw waveform features yields the highest accuracy, with Mel spectrogram being the most influential. Experimental results on the IEMOCAP and RAVDESS databases achieve emotion recognition accuracy of 83.2% and 97.8%, respectively, showcasing improved performance and robustness compared to state-of-the-art models, while using fewer trainable parameters.