Multi-Prototype Grouping for Continual Learning in Visual Question Answering

Abstract

Visual Question Answering (VQA) aims to answer questions utilizing information from both textual and visual modalities. New data categories and novel combinations of the two modalities will continuously emerge in practical applications, necessitating continual learning. For this unique compositional generalization challenge, existing work adopts prototype learning to separately capture sample-invariant features of question types and image objects, retrieving the most similar prototype from two modalities as generalizable representations. However, for semantic fuzzy samples that deviate from the corresponding prototype, relying on a single prototype may lead to error-prone representation learning. Additionally, for the catastrophic forgetting challenge, existing methods employ memory to store past samples but lack an effective selection of samples to be stored, merely updating the memory through random sampling. In this paper, we propose ProtoGroup, a multi-prototype grouping continual learning framework for VQA, which groups prototypes based on their similarity to obtain more accurate and stable sample-invariant features. We also devise a memory sample selection method based on the correlation with prototype groups from old tasks. Experimental results demonstrate our framework outperforms previous methods significantly across multiple datasets and settings.