Prototypical Graph Alignment for Text-based Person Search

Abstract

Text-based Person Search is one of the downstream tasks of cross-modal retrieval. The key challenge is aligning features of two extremely irrelavant modalities into the same latent space. Recent works within Prototype Learning introduce a few learnable parameters to map heterogeneous features into the same prototype space, aligning them using weighted-sum prototype on global matching or independent prototypes on local matching. However, these matches both ignore the relationship between prototypes. Thus we propose a Prototypical Graph Alignment (PGA) architecture for Text-based Person Search. For a given mini-batch, our PGA first transforms image-text features of identical person to his corresponding prototype space, enabling unique individual to obtain customized unimodal prototypical graphs and cross-modal fused graphs, and each graph consists of prototype representation and comparable relationship. By minimizing the discrepancy between unimodal and cross-modal graphs, we can achieve node-level local alignment and edge-level relational alignments. Extensive experiments conducted on three primary TPS datasets demonstrate the effectiveness of the proposed method.