Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning
Bo Wan, Yongfei Liu, Desen Zhou, Tinne Tuytelaars, Xuming He
Abstract
Human object interaction (HOI) detection plays a crucial role in human-centric scene understanding and serves as a fundamental building block for many vision tasks. One generalizable and scalable strategy for HOI detection is to use weak supervision, learning from image-level annotations only. This is inherently challenging due to ambiguous human-object associations, large search space of detecting HOIs and highly noisy training signal. A promising strategy to address those challenges is to exploit knowledge from large-scale pretrained models (e.g., CLIP), but a direct knowledge distillation strategy does not perform well on the weakly-supervised setting. In contrast, we develop a CLIP-guided HOI representation capable of incorporating the prior knowledge at both image level and HOI instance level, and adopt a self-taught mechanism to prune incorrect human-object associations. Experimental results on HICO-DET and V-COCO show that our method outperforms the previous works by a sizable margin, showing the efficacy of our HOI representation.
BibTeX
@inproceedings{
wan2023weaklysupervised,
title={Weakly-supervised {HOI} Detection via Prior-guided Bi-level Representation Learning},
author={Bo Wan and Yongfei Liu and Desen Zhou and Tinne Tuytelaars and Xuming He},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=resApVNcqSB}
}