DS-BTIAN: A Novel Deep-Shallow Bidirectional Transformer Interactive Attention Network for Multimodal Emotion Recognition

Abstract

In this work, we propose a novel Deep-Shallow Bidirectional Transformer Interactive Attention Network (DS-BTIAN) designed for robust multimodal emotion recognition. DS-BTIAN leverages pre-trained Wav2Vec2.0 and BERT models for efficient feature extraction without the need for finetuning, enhancing representation by integrating outputs from both shallow and deep layers. A novel bidirectional cross-modal interactive attention mechanism based on multi-head attention is introduced, adaptively modeling the relationship between speech and text, thus promoting effective multimodal fusion. Additionally, we incorporate an adaptive speech segmentation and emotion aggregation strategy to better capture emotional dynamics. The network is optimized using a joint training strategy combining center contrastive loss and angular Softmax loss, enhancing discriminative feature learning. We conducted extensive speaker-independent ten-fold cross-validation experiments on the IEMOCAP dataset. The results demonstrate that DS-BTIAN significantly outperforms state-of-the-art methods, achieving a remarkable average weighted accuracy of 76.37% and an unweighted accuracy of 77.21% on the test set, representing a substantial improvement over baseline models. Ablation studies further confirm the effectiveness of each component, underscoring DS-BTIAN’s outstanding capability in capturing and integrating multimodal emotional information. By utilizing pre-trained models for feature extraction without fine-tuning, our approach not only delivers superior performance but also offers a resource-efficient solution, particularly valuable in scenarios with limited computational resources or annotated data.