Memory Self-Calibrated Network for Visual Grounding

Abstract

Visual Grounding (VG) aims to locate the most relevant object or region in an image according to a natural language query. Existing methods in VG utilize fixed image and text representations to capture cross-modal semantic consistency, which limits the flexibility in adjusting image representations according to diverse textual information and hinders performance. To handle this limitation, we propose a novel Memory Self-Calibrated Network (MSCN) by dynamically refining image representations based on the query, thereby improving the semantic consistency between texts and images for visual grounding. Specifically, we introduce two modules: Semantic Relevance Filtering Module (SRFM) and Adaptive Memory Fusion Module (AMFM), to explicitly model the relationship between image and text. SRFM focuses on filtering out image information that is irrelevant to the query, while AMFM adaptively fuses text-related representations with initial image features to enhance the understanding ability of the MSCN model. Comprehensive experiments on three datasets demonstrate the superiority of our method compared to existing approaches.