Sound-Indicated Visual Object Detection for Robotic Exploration

Feng Wang; Di Guo; Huaping Liu; Junfeng Zhou; Fuchun Sun

ICRA 2019poster9 citations

Sound-Indicated Visual Object Detection for Robotic Exploration

Feng Wang, Di Guo, Huaping Liu, Junfeng Zhou, Fuchun Sun

Abstract

Robots are usually equipped with microphones and cameras to perceive and understand the physical world. Though visual object detection technology has achieved great success, the detection in other modalities remains unsolved. In this paper, we establish a novel robotic sound-indicated visual object detection framework, and develop a two-stream weakly-supervised deep learning architecture to connect the visual and audio modalities for localizing the sounding object. A dataset is constructed from the AudioSet to validate the proposed method and some promising applications are demonstrated on robotic platforms.

BibTeX

@inproceedings{icra2019_soundindicatedvi,
  title = {Sound-Indicated Visual Object Detection for Robotic Exploration},
  author = {Feng Wang and Di Guo and Huaping Liu and Junfeng Zhou and Fuchun Sun},
  booktitle = {ICRA 2019},
  year = {2019}
}