Deeply Coupling EEG Signals and Eye Movements for Multi-Modal and Region-Aware Emotion Recognition

Abstract

Automatic emotion recognition based on electroencephalogram (EEG) signals has been a significant clinical approach to detect emotional states. Given the intuitive complementation between physiological signals and behavioral signals, combining EEG signals with facial expressions, e.g., eye movements, should be a promising direction to further robustness and performance of emotion recognition methods. However, existing deep learning-based multi-modal methods tend to fuse the multi-modal features only in the last layer of the networks, which is too explict and shallow to deeply mine the implicit knowledge complementarity between the modalities. Additionally, existing methods fail to fully consider the distribution diversity of brain topology for different emotion states, which deserves further adaptive modeling. In this paper, we propose DCEE, a novel multi-modal and region-aware emotion recognition method which Deeply Couples EEG signals and Eye movements with a self-attention mechanism-based inter and intra-modal feature fusion module, and realizes adaptive modeling of brain activation patterns based on our designed emotion-region correlation matrix. Extensive experiments conducted on datasets SEED-IV and DEAP demonstrate the effectiveness of our proposed method and the complementation between EEG signals and eye movements for emotion recognition.