Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching
Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, Dejing Dou
Abstract
Discriminatively localizing sounding objects in cocktail-party, i.e., mixed sound scenes, is commonplace for humans, but still challenging for machines. In this paper, we propose a two-stage learning framework to perform self-supervised class-aware sounding object localization. First, we propose to learn robust object representations by aggregating the candidate sound localization results in the single source scenes. Then, class-aware object localization maps are generated in the cocktail-party scenarios by referring the pre-learned object knowledge, and the sounding objects are accordingly selected by matching audio and visual object category distributions, where the audiovisual consistency is viewed as the self-supervised signal. Experimental results in both realistic and synthesized cocktail-party videos demonstrate that our model is superior in filtering out silent objects and pointing out the location of sounding objects of different classes. Code is available at https://github.com/DTaoo/Discriminative-Sounding-Objects-Localization.
BibTeX
@inproceedings{NEURIPS2020_7288251b,
author = {Hu, Di and Qian, Rui and Jiang, Minyue and Tan, Xiao and Wen, Shilei and Ding, Errui and Lin, Weiyao and Dou, Dejing},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {10077--10087},
publisher = {Curran Associates, Inc.},
title = {Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/7288251b27c8f0e73f4d7f483b06a785-Paper.pdf},
volume = {33},
year = {2020}
}