Multimodal Graph-Based Audio-Visual Event Localization

Abstract

The task of audio-visual event localization requires a model that can identify consistent events between audio and visual data in a video, where the two types of data may conflict or be obscured by background noise. Previous methods for audio-visual event analysis used features from all areas of frames and audio to establish relationships between them for matching. However, within different frames of the same video, most regions are similar, with variations occurring only in certain specific areas. To identify these crucial regions, we introduce a multimodal fusion approach based on graph neural networks. This involves constructing and processing graph using patchified images and audio to find the key patches essential for prediction. Our experimental results validate that our model achieves superior performance compared to previous state-of-the-art methods, even with fewer parameters on the AVE benchmark dataset.