Relaxing Distillation Constraints for Improved New Class Learning in Continual Semantic Segmentation

Abstract

Continual Semantic Segmentation (CSS) aims to continuously learn new classes while mitigating catastrophic forgetting. Existing CSS methods primarily address this challenge through knowledge distillation. While they focus on maintaining stability for old classes, this emphasis often restricts plasticity in learning new ones. In this work, we propose two plug-and-play methods to better balance stability and plasticity. First, for feature map distillation, we replace pixel-level constraints with statistical information extraction within a window, improving the learning of new classes while preserving structural integrity to overcome forgetting. Second, we use relative ranks of predictions to replace exact probabilistic values to loosen the constraints in the logit distribution of old classes. Extensive experiments demonstrate that our methods outperform state-of-the-art methods when integrating into several baselines across various CSS scenarios.