RefCap: Zero-shot Video Corpus Moment Retrieval Based on Refined Dense Video Captioning

Abstract

Video corpus moment retrieval (VCMR) is a challenging task aimed at localizing specific segments from untrimmed videos within a vast video collection. It has long been addressed using end-to-end supervised or weakly-supervised methods, which often lack explainability and rely on laborious annotations. To address these issues, we exploit generated captions from Vision Large Language Models (VLLMs) and propose the first zero-shot training-free VCMR system, RefCap. The system consists of two decoupled stages: the Construction Stage to propose dense events and construct corresponding captions, and the Retrieval Stage to retrieve events based on text similarities between queries and captions. In the Construction Stage, we design a Sliding-Window Denoiser and a Quality-Measured Event Generator to produce high-quality dense captions, and derive the Indexing Keyword Sets to fully utilize the generated captions. In the Retrieval Stage, we propose a multi-granularity retrieval strategy integrating both sentence-level and word-level textual similarities between queries and event captions. Extensive experiments on the Charades and ActivityNet datasets demonstrate the effectiveness and competitiveness of our method. Code is available at https://github.com/BUAAPY/RefCap.