One-Shot Medical Action Recognition With A Cross-Attention Mechanism And Dynamic Time Warping

Abstract

In this paper, we address the classification of medical actions with only one single sample by developing a novel one-shot learning framework which contains both cross-attention and dynamic time warping (DTW) modules. To be concrete, we firstly transform the raw skeleton sequence into the signal-level image representation. We exploit a metric learning approach, which is the prototypical network for the proposed one-shot learning framework and choose the residual network (ResNet18) as the backbone which is widely used in recent years. Cross-attention is applied for guiding the network to focus on the more important joints from each specific action. The cross-attention mechanism that applies between the support and query set will be adapted for mining and matching the relationships with the human body. Furthermore, a DTW module is introduced to mitigate the temporal information mismatching issue between the actions from the support and query sets. The experimental results on the NTU RGB+D 120 dataset demonstrate the effectiveness of our proposed approach and the improved performance compared to the baseline approach. The code of this work is available at <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .