Hybrid Graph Convolutional Networks for Skeleton-Based and EEG-Based Jumping Action Recognition

Abstract

Kinematic information obtained directly from the skeletal model has been useful for jumping action recognition. Current research focuses on dynamic analysis based on the video stream. Although skeletal data can accurately capture the high-level information of human action, it ignores the brain’s pre-execution command information, which plays a crucial role in identifying jumping action. Therefore, we proposed a hybrid model based on brain network and dynamic skeleton. Specifically, we used a brain network graph convolutional network (BNGCN) to encode brain command information. Also, a dynamic skeleton convolutional network (DSGCN) using the angular velocity of skeleton nodes instead of video is proposed, which can break the fixed experimental area’s limitation. BNGCN and DSGCN are fused through three network nodes to construct an end-to-end Brain Network and Dynamic Skeleton Hybrid Model. Our contribution consists of three parts. First, we have created a data set that can be used for jumping action and its sub-phase recognition. Second, BNGCN is used to extract brain command information for jumping action recognition. Third, a hybrid model is proposed to incorporate brain command and skeleton kinematic information. The results show that our hybrid model can effectively capture the high-level features for jumping action recognition. The method outperforms compared methods for jumping action recognition.