Enhancing Video-Text Matching via Sparse Stratified Sampling

Abstract

Video-text matching is a critical task in multimedia retrieval, but traditional methods often fail to capture the diversity and depth of video content due to inefficient and inaccurate frame sampling. We propose a novel sparse stratified sampling technique that can substantially improve the video-text matching process by segmenting video content into clusters based on relevant features and selectively sampling representative frames. Our method further introduces a threshold for the feature metric used to divide clusters, eliminating video frames with low relevance. We propose two variants of our approach: an offline approach that performs sampling before training, and an online approach that dynamically conducts sampling based on the relevance between video frames and the text query during training. Extensive experiments on datasets like MSRVTT and AVSD for video retrieval and multiple-choice VideoQA datasets, including AVQA and Music-AVQA, demonstrate the superiority of our method over previous state-of-the-art approaches. Our sparse stratified sampling technique achieves improvements of over 1.2% on MSRVTT and 1.7% on AVSD for R@1 in video retrieval tasks. For multiple-choice VideoQA tasks, our approach achieves significant improvements of 1.8% accuracy on AVQA and 3.9% on Music-AVQA, strongly supporting its effectiveness in enhancing video-text matching systems.