RECANTFormer: Referring Expression Comprehension with Varying Numbers of Targets
Bhathiya Hemanthage, Hakan Bilen, Phil Bartie, Christian Dondrup, Oliver Lemon
Abstract
The Generalized Referring Expression Comprehension (GREC) task extends classic REC by generating image bounding boxes for objects referred to in natural language expressions, which may indicate zero, one, or multiple targets. This generalization enhances the practicality of REC models for diverse real-world applications. However, the presence of varying numbers of targets in samples makes GREC a more complex task, both in terms of training supervision and final prediction selection strategy. Addressing these challenges, we introduce RECANTFormer, a one-stage method for GREC that combines a decoder-free (encoder-only) transformer architecture with DETR-like Hungarian matching. Our approach consistently outperforms baselines by significant margins in three GREC datasets.
BibTeX
@inproceedings{hemanthage-etal-2024-recantformer,
title = "{RECANTF}ormer: Referring Expression Comprehension with Varying Numbers of Targets",
author = "Hemanthage, Bhathiya and
Bilen, Hakan and
Bartie, Phil and
Dondrup, Christian and
Lemon, Oliver",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-main.1214/",
doi = "10.18653/v1/2024.emnlp-main.1214",
pages = "21784--21798"
}