Hybrid Spatial-Frequency Attention Network For Fine-Grained Skeleton-Based Action Recognition

Abstract

Recently, Transformer-based methods have gained popularity in skeleton-based action recognition due to their advantages in modeling long-range dependencies. However, Transformer lacks the inductive biases towards skeletal topology and tends to capture salient features, potentially overlooking subtle inter-class variations in similar actions, leading classifications. To address these issues, we first propose Hybrid Spatial-Frequency Attention Network (HSFA-Net) with spatial and temporal structure for fine-grained action recognition. The spatial structure includes a Frequency Domain Channel Enhancement module, which utilizes the Discrete Cosine Transform to convert grouped skeleton data into frequency domain and refines it with multiple high-frequency components. Next, we propose Spatial-Frequency Attention Fusion module to integrate spatial and frequency features for a more comprehensive feature representation. In the temporal structure, we apply the concept of temporal differences to human skeletons and design a Temporal Motion-Sensitive module to highlight motion-sensitive features. Finally, we propose Temporal Frequency Attention module for richer temporal features in the frequency domain. Extensive experiments on NTU-RGB+D and NTU-RGB+D 120 datasets validate the effectiveness of HSFA-Net.