Vision-Sensor Attention Based Continual Multimodal Egocentric Activity Recognition

Abstract

Continual learning aims to equip deep neural networks (DNNs) with the capability to continuously learn new knowledge without catastrophic forgetting. Currently, there is significant attention on multimodal continual activity recognition from a egocentric perspective. However, the issue of modality imbalance can lead to exacerbated forgetting in multimodal continual learning. To address this, we propose an exemplar-free vision-sensor Attention-based Incremental Discriminability enhancement (AID) method. Firstly, we employ a Vision-Sensor attention module to enhance the time-frequency information of sensor modality and fuse them with vision modality. This alleviates the modality imbalance problem, yielding more discriminative and generalizable representations. Simultaneously, to prevent the classifier from overfitting to old class prototypes, we enhance old prototypes with features from new classes, thereby enhancing classifier discriminability. We validate the effectiveness of this method through numerous experiments with various task settings on the UESTC-MMEA-CL dataset.