Urban Sound & Sight: Dataset And Benchmark For Audio-Visual Urban Scene Understanding
Magdalena Fuentes, Bea Steers, Pablo Zinemanas, Martín Rocamora, Luca Bondi, Julia Wilkins, Qianyi Shi, Yao Hou
Abstract
Automatic audio-visual urban traffic understanding is a growing area of research with many potential applications of value to industry, academia, and the public sector. Yet, the lack of well-curated resources for training and evaluating models to research in this area hinders their development. To address this we present a curated audio-visual dataset, Urban Sound & Sight (Urbansas), developed for investigating the detection and localization of sounding vehicles in the wild. Urbansas consists of 12 hours of unlabeled data along with 3 hours of manually annotated data, including bounding boxes with classes and unique id of vehicles, and strong audio labels featuring vehicle types and indicating off-screen sounds. We discuss the challenges presented by the dataset and how to use its annotations for the localization of vehicles in the wild through audio models.
BibTeX
@inproceedings{icassp2022_urbansoundsightd,
title = {Urban Sound & Sight: Dataset And Benchmark For Audio-Visual Urban Scene Understanding},
author = {Magdalena Fuentes and Bea Steers and Pablo Zinemanas and Martín Rocamora and Luca Bondi and Julia Wilkins and Qianyi Shi and Yao Hou and Samarjit Das and Xavier Serra and Juan Pablo Bello},
booktitle = {ICASSP 2022},
year = {2022}
}