FCC-MF: Detecting Violence in Audio-Visual Context with Frame-Wise Cluster Contrast and Modality-Stage Flooding

Abstract

This paper explores the detection of frame-wise instances of violence in both audio and visual modalities, where only clip-level labels are available. Previous works selected fixed value of frames for objective optimization to model frame-level features, and applied straightforward fusion strategy to aggregate audio and visual information. However, these two issues, namely Constant Frames Selection and Vulnerable Fusion, significantly impair the network’s detection performance. To address these issues, we present a novel framework called Frame-wise Cluster Contrast with Modality-stage Flooding (FCC-MF). Our contributions include: 1) We propose Frame-wise Cluster Contrast, which leverages unsupervised clustering for pseudo-labeling frames and triplet loss for contrastive learning to allow for dynamic frame-wise discrimination. 2) We propose Modality-stage Flooding, a two-stage flooding approach with the higher loss flooding level assigned to uni-modal features, which prevents over-memorization of redundant uni-modal data and promotes effective aggregation of multi-modal information. Our FCC-MF framework yields a promising average precision of 84.24% on the XD-Violence dataset, which performs favorably against previous SOTA methods. Extensive ablation studies exhibit that our FCC-MF framework produces finer frame-level violence discrimination ability and generalizable audio-visual fusion features.