SGPT: The Secondary Path Guides the Primary Path in Transformers for HOI Detection

Abstract

HOI detection is essential for human-computer interaction, especially in behavior detection and robot manipulation. Existing mainstream transformer methods of HOI detection are focused on single-stream detection only, e.g., image \rightarrow HOI(\mathcal{P}_{1})image \rightarrow HOI(\mathcal{P}_{1}), or image \rightarrow HO\rightarrow I(\mathcal{P}_{2})image \rightarrow HO\rightarrow I(\mathcal{P}_{2}). Both paths have their own characteristics of concern, so we propose a novel method, using the Secondary path (\mathcal{P}_{2})(\mathcal{P}_{2}) Guides the Primary path (\mathcal{P}_{1})(\mathcal{P}_{1}) in Transformers (SGPT). SGPT contains two core modules: the Dual-Path Consistency (DPC) module and the Instance Interaction Attention (IIA) module. DPC keeps human, object and interaction consistent on the dual-path and lets \mathcal{P}_{2}\mathcal{P}_{2} guide \mathcal{P}_{1}\mathcal{P}_{1} to learn more meaningful features. IIA fuses human and object to enhance interaction in \mathcal{P}_{2}\mathcal{P}_{2}, which allows instance to constrain interaction. Our proposed dual-path are employed during training, and only the \mathcal{P}_{1}\mathcal{P}_{1} path is used for inference. Hence, SGPT improves generalization without increasing model capacity in HICO-DET and V-COCO datasets compared to the state-of-the-arts. The code of this work is available at https://github.com/visualVk/sgpt.git.