Audio-Visual Tracking of Multiple Speakers Via a PMBM Filter

Jinzheng Zhao; Peipei Wu; Xubo Liu; Yong Xu; Lyudmila Mihaylova; Simon J. Godsill; Wenwu Wang

ICASSP 2022accepted0 citations

Audio-Visual Tracking of Multiple Speakers Via a PMBM Filter

Jinzheng Zhao, Peipei Wu, Xubo Liu, Yong Xu, Lyudmila Mihaylova, Simon J. Godsill, Wenwu Wang

Abstract

Audio-visual tracking of multiple speakers requires to estimate the state (e.g. velocity and location) of each speaker by leveraging the information of both audio and visual modalities. Estimating the number of speakers and their states jointly remains a challenging problem. We propose an Audio-Visual Possion Multi-Bernoulli Mixture Filter (AV-PMBM) that can not only predict the number of speakers but also give accurate estimation of their states. We also propose a novel sound source localization technique based on DOA information and a deep learning based object detector to provide reliable audio measurements for the AV tracker. To our knowledge, this represents the first attempt using PMBM for multi-speaker tracking with audio visual modalities. Experiments on the AV16.3 dataset demonstrate that AV-PMBM achieves state-of-the-art performance in optimal sub-pattern assignment (OSPA).

BibTeX

@inproceedings{icassp2022_audiovisualtrack,
  title = {Audio-Visual Tracking of Multiple Speakers Via a PMBM Filter},
  author = {Jinzheng Zhao and Peipei Wu and Xubo Liu and Yong Xu and Lyudmila Mihaylova and Simon J. Godsill and Wenwu Wang},
  booktitle = {ICASSP 2022},
  year = {2022}
}