FAWL: Weakly-Supervised Video Corpus Moment Retrieval with Frame-Wise Auxiliary Alignment and Weighted Contrastive Learning

Abstract

Video Corpus Moment Retrieval (VCMR) is a challenging task that aims to localize query-specified moments from a collection of untrimmed videos. The recent state-of-the-art method, JSG, tries to tackle this task using only video-level annotations in a weakly-supervised setting. However, the late fusion strategy of JSG suffers from insufficient alignment, and the proposal-level video representations aggregated with Gaussian Distributions result in semantic inconsistency across frames. To address these issues, we propose a novel weakly-supervised VCMR method, FAWL, which incorporates frame-wise auxiliary alignment and weighted contrastive learning. Two frame-wise auxiliary alignment tasks, namely Query-guided Saliency Alignment (QSA) and Event-aware Boundary Alignment (EBA), are designed first. During training, QSA projects video and text representations through a shared layer, and computes frame-level saliency losses for sufficient multimodal alignment. EBA guides frames to predict distances to proposal boundaries making each frame aware of the corresponding events and thus ensuring semantic consistency. To help the model distinguish proposals with similar visual semantics, we further propose the Weighted Contrastive Learning (WCL) which integrates inter-video similarities into the conventional InfoNCE loss. FAWL enjoys both improved alignment and inference efficiency, achieving new state-of-the-art performance on two challenging datasets. Code is available at https://github.com/BUAAPY/FAWL.