Heterogeneous Graph Convolutional Neural Networks for EEG-fNIRS Bimodal Emotion Recognition

Abstract

Leveraging multimodal brain signals, such as electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS), for the objective detection of brain activity is regarded as a promising approach for affective brain-computer interface. Existing EEG-fNIRS bimodal methods primarily focus on data alignment and basic feature fusion, neglecting the intrinsic connections between different signals and brain regions. To this end, we propose a novel graph-based method, the Heterogeneous Graph Convolutional Neural Network (HGCN), which integrates multimodal complementary information and constructs a heterogeneous EEG-fNIRS interaction graph within a coordinated hyperspace to model brain networks. Specifically, various directed edges types and node feature aggregation strategies are employed to dynamically update and enhance the representation of brain signals and emotional states, thereby improving the coherence and consistency of cross-modal signals. Furthermore, this approach provides a robust framework for exploring cross-modal spatial connectivity. In this paper, we also developed a novel EEG-fNIRS emotion database collected from 17 subjects by video stimuli. Extensive experimental results demonstrate the superiority of our method and the effectiveness of the introduced heterogeneous EEG-fNIRS interaction graph.