Efficient Prototypical Classifier for Class-Incremental Learning

Abstract

The nearest prototypical classifier faces challenges of semantic drift and prototype interference. Previous methods address these issues using data rehearsal and contrastive learning, but these approaches incur high memory costs and slow convergence. In this paper, we propose a novel prototypical minimum distance loss, along with a two-stage training pipeline, to mitigate prototype interference with low memory overhead and fast convergence. Leveraging task-specific prompts and a key-query mechanism, we significantly reduce semantic drift. Additionally, we introduce a continual exponential moving average to enhance model stability and minimize forgetting. Notably, our method is rehearsal-free and avoids generation processes, simplifying training and further reducing memory usage. We validate our approach on four challenging class-incremental learning datasets, achieving significant improvements over state-of-the-art methods.