Anchor-Prompt-based Segmentation and Embedding Model

Abstract

Tackling multi-object tracking and segmentation (MOTS) can be attributed to a multi-task learning task, i.e., performing Segmentation and Identity Embedding jointly (SIEJ). Unfortunately, achieving optimal SIEJ is non-trivial, as it relies on different spatiotemporal features of objects. Besides, the lack of labeled data raises the difficulty of balancing the two objectives in SIEJ. Empowered by the Segment Anything Model (SAM), we propose an Anchor-Prompt-based Segmentation and Embedding Model (APSEM) towards optimal SIEJ by introducing redundant anchors and designing an embedding decoder. On the one hand, our APSEM allows redundant anchors for the same object to identify occluded objects and refine the pixel-wise edge mask. On the other hand, our proposed embedding decoder is designed to address identity learning for redundant anchors by minimizing differences between the same identities in redundant prompts. We also create a parameter-efficient fine-tuning strategy, which helps combine the embedding module into the foundation model through a bit of data. Experimental results on MOTSChallenge datasets validate the effectiveness of the proposed APSEM method for MOTS tasks. Such results also demonstrate that each module improves segmentation and identity embedding performance through joint training.