Visual Prompt Tuning for Weakly Supervised Phrase Grounding
Pengyue Lin, Zhihan Yu, Mingcong Lu, Fangxiang Feng, Ruifan Li, Xiaojie Wang
Abstract
Previous works on the task of weakly supervised phrase grounding (WSG) rely heavily on object detectors providing RoIs for the localization. However, such methods cannot be applied effectively to real-world scenarios largely because that the detectors are trained with limited categories. In this paper, we propose a refinement-based approach to WSG through fine-tuning a detector-free phrase grounding model with a visual prompt. This visual prompt is extracted from the text-related representations in CLIP. Furthermore, we combine the visual prompt with learnable features and then fine-tune the grounding network. Our experimental results significantly outperform state-of-the-art methods on the WSG task and shows the effectiveness of our method.
BibTeX
@inproceedings{icassp2024_visualprompttuni,
title = {Visual Prompt Tuning for Weakly Supervised Phrase Grounding},
author = {Pengyue Lin and Zhihan Yu and Mingcong Lu and Fangxiang Feng and Ruifan Li and Xiaojie Wang},
booktitle = {ICASSP 2024},
year = {2024}
}