PASTD: Progressive Augmentation and Spatiotemporal Decoupling Contrastive Learning for Skeleton-Based Action Recognition

Abstract

Contrastive learning has achieved significant progress in the field of self-supervised skeleton-based action recognition. However, existing methods often apply strong augmentations directly to skeleton data, which can distort or even lose the semantic of the skeletons. Additionally, most methods focus on unified feature extraction through spatiotemporal modeling, leading to spatiotemporal entanglement that hinders model’s interpretability and degrades performance. To address these issues, we propose a Progressive Augmentation and SpatioTemporal Decoupling contrastive learning model (PASTD). Specifically, we propose progressive augmentation to enhance the model’s generalization capability, which generates multiple distinct positive sample pairs through multiple branches and uses inter-branch constraints to maintain consistent motion patterns. Moreover, we propose spatiotemporal decoupling module to separate the feature’s spatial and temporal information, using a dual-path self-attention module combined with an intra-branch cross-domain learning strategy to facilitate information exchange between domains. Extensive experiments on the four public datasets NTU-RGB+D (60&120) and PKU-MMD (I&II) demonstrate the effectiveness of these components. Moreover, PASTD outperforms state-of-the-art methods across various evaluation metrics.