EmoTVR: A Hybrid Model to Estimate Continuous-Time and Continuous-Level Emotion from Electroencephalography

Abstract

Emotion recognition from electroencephalography (EEG) has attracted widespread interest, but few studies have considered estimating the highly dynamic trajectories of emotion in a relatively long period, such as video watching. To address this problem, we first recruit participants to assign continuous-time and continuous-level emotion labels to videos from the SEED corpus. Then, we propose a hybrid model, namely Emotion Time-Varying Regression (EmoTVR), to estimate continuous-time and continuous-level emotion using EEG spatial-temporal representations. EmoTVR combines atrous convolutional networks for spatial feature extraction and a temporal self-attentive regressor using attention-based long short-term memory for temporal feature extraction and continuous estimation. Moreover, EmoTVR adopts the Domain Adversarial Neural Network to address the problem of individual difference. Experimental results through both within-subject and cross-subject cross-validations demonstrate the superiority of EmoTVR in recognizing dynamic and continuous emotion over traditional methods. The proposed EmoTVR method caters for the needs of dynamic and continuous emotion recognition in naturalistic conditions, so it is highly potential for practical applications of emotion recognition.