Improving Open-Ended Referring Expression Comprehension via Dual-Language Constraints

Abstract

Open-ended referring expression comprehension focuses on locating the text query within an image via scene knowledge, requiring complex reasoning across the triplet of the image, scene knowledge, and the text query. However, most existing methods struggle to integrate scene knowledge while performing a single-object prediction. To address this issue, we propose VG-DLC, a model that progressively uses scene knowledge and open-ended referring expressions as constraints for reasoning and grounding. Specifically, we first align scene knowledge with the referring expression and the image in sequence, which supports effective relational reasoning and implicitly constrains the open-ended content to corresponding image regions. Next, we explicitly align the referring expression with the image and employ it as a query vector, constraining it to a unique region to achieve a single end-to-end prediction. Extensive experiments on the SK-VG dataset validate the effectiveness of our method, demonstrating that VG-DLC outperforms existing end-to-end approaches.