Eigen and multimodal analysis for localizing moving sounding objects
Shreya Khare, Akshay Bhandari, Hema A. Murthy
Abstract
This paper identifies moving objects in a video that are associated to the corresponding audio, by exploiting the correlation of audio and video features. The proposed technique is based on the correlation of motion features of eigen moving objects with audio mel frequency cepstral coefficients features using canonical correlation analysis. We propose two strategies to detect the eigen moving objects: (i) Per-frame mapped eigen moving object (PFEMO) and (ii) Temporally coherent eigen moving object (TCEMO). While PFEMO segments each frame using superpixel segmentation, TCEMO exploits supervoxel based video segmentation to identify eigen moving objects. Qualitative (mean-opinion score) and quantitative (precision, recall, area under the curve, hit ratio) analysis shows that the performance of the proposed techniques is superior to those of the state-of-the-art methods.
BibTeX
@inproceedings{icassp2016_eigenandmultimod,
title = {Eigen and multimodal analysis for localizing moving sounding objects},
author = {Shreya Khare and Akshay Bhandari and Hema A. Murthy},
booktitle = {ICASSP 2016},
year = {2016}
}