Enhancing Continual Learning for Medical Imaging: Efficient Knowledge Transfer and Multi-Disease Prediction

Abstract

Deep learning models for medical disease detection require extensive labeled data, which is often scarce and expensive. Transfer learning can help by leveraging knowledge from large source domains, but directly fine-tuning these models can lead to catastrophic forgetting, making it impossible to reuse it for new diseases that continue to emerge. Current continual learning methods can mitigate catastrophic forgetting but fail to support efficient knowledge transfer and use mutually exclusive classifiers, which are inadequate for multi-disease prediction in medical imaging. To solve these issues, we propose a Continual Learning Multi-Disease Prediction Framework with class-specific adapters to independently model each disease, supporting multi-disease prediction and feature integration. Additionally, we introduce a two-stage knowledge integration method based on self-attention, which concatenates features from previous tasks and computes attention scores to utilize historical knowledge effectively. Experimental results demonstrate that our framework supports multi-disease prediction and knowledge transfer effectively. They also confirm the effectiveness of our self-attention-based method, providing robust support for medical disease detection.