Unsupervised Open-Vocabulary Object Localization in Videos
Ke Fan, Zechen Bai, Tianjun Xiao, Dominik Zietlow, Max Horn, Zixu Zhao, Carl-Johann Simon-Gabriel, Mike Zheng Shou
Abstract
In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization. We propose a method that first localizes objects in videos via a slot attention approach and then assigns text to the obtained slots. The latter is achieved by an unsupervised way to read localized semantic information from the pre-trained CLIP model. The resulting video object localization is entirely unsupervised apart from the implicit annotation contained in CLIP, and it is effectively the first unsupervised approach that yields good results on regular video benchmarks.
BibTeX
@inproceedings{iccv2023_unsupervisedopen,
title = {Unsupervised Open-Vocabulary Object Localization in Videos},
author = {Ke Fan and Zechen Bai and Tianjun Xiao and Dominik Zietlow and Max Horn and Zixu Zhao and Carl-Johann Simon-Gabriel and Mike Zheng Shou and Francesco Locatello and Bernt Schiele and Thomas Brox and Zheng Zhang and Yanwei Fu and Tong He},
booktitle = {ICCV 2023},
year = {2023}
}