Pseudo-Query Generation For Semi-Supervised Visual Grounding With Knowledge Distillation

Abstract

Visual grounding is a crucial multi-modal job for locating the objects that the referring queries refer to in images. In recent years, both fully-supervised and weakly-supervised algorithms rely on a large number of query annotations. However, collecting queries in natural language is labor-intensive, which limits the application scenarios of these methods. To overcome this weakness, we propose a novel semi-supervised visual grounding framework. The framework consists of two effective techniques: a prompt enhanced pseudo-query generator utilizing objects without query annotation to produce high-quality pseudo-queries; a knowledge distillation mechanism using a teacher network to stabilize the training process of the student network. Experiment results show that our proposed framework dramatically outperforms the existing methods under the three different label proportions on the three commonly used visual grounding datasets.