Towards Differential Optimization: Rehearsal-Free Class-Incremental Learning with Slow Learners and Fast Adapters

Abstract

Class-incremental learning (CIL) enables models to learn new tasks without forgetting previously acquired knowledge. However, existing CIL approaches often struggle with inadequate adaptation to task-specific feature spaces and catastrophic forgetting of previously-acquired knowledge, compromising the models’ plasticity and stability. To address these challenges, this paper proposes a novel differential optimization paradigm called DO-CIL, which incorporates task-agnostic slow learner (TSL) with task-specific fast adapter (TFA) for rehearsal-free CIL. Specifically, TSL aims to effectively capture shared knowledge with low learning rates for robust generalization, while TFA allows pre-trained models to adapt to new task-specific feature spaces. Benefitting from the classifier retraining strategy, a learnable semantic shift network is also proposed to align prototypes with the evolving model representation, facilitating the retraining of task-specific classifiers based on these updated prototypes. Extensive experiments on multiple benchmark datasets consistently demonstrate the superiority and effectiveness of our DO-CIL approach compared to state-of-the-art baselines.