Dynamic Category Queries Transformer for Generalized Few-shot Semantic Segmentation

Abstract

Few-shot segmentation (FSS) tackles data scarcity using multiple priors, but its simplicity limits handling base and novel classes with limited data access. Generalized few-shot semantic segmentation (GFSS) enhances model performance for base classes with abundant data, while novel classes have limited data access, improving generalization with scarce data. Building on the design of query-based segmentation models, which decouple the mask and classification tasks for individual optimization, we here present the Dynamic Category Queries Transformer (DCQ-Former) which forms a novel approach to the GFSS. The proposed DCQ-Former first uses category suggested dynamic queries to perform mask segmentation and category classification tasks on a large amount of base class data. Considering the case when the novel classes only have access to a limited amount of training data, the queries for the novel classes are instead dynamically composed from the base classes in order to prevent the category suggested module from providing limited suggestion queries given the representativeness of the fewshot samples. Extensive experiments on COCO-20<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and Pascal-5<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> datasets show that DCQ-Former achieves superior accuracy and generalization than current state-of-the-art methods. Our code are available at https://github.com/fallpavilion/DCQ-Former.