Investigations on End- to-End Audiovisual Fusion

Michael Wand; Jürgen Schmidhuber; Ngoc Thang Vu

ICASSP 2018accepted0 citations

Investigations on End- to-End Audiovisual Fusion

Michael Wand, Jürgen Schmidhuber, Ngoc Thang Vu

Abstract

Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.

BibTeX

@inproceedings{icassp2018_investigationson,
  title = {Investigations on End- to-End Audiovisual Fusion},
  author = {Michael Wand and Jürgen Schmidhuber and Ngoc Thang Vu},
  booktitle = {ICASSP 2018},
  year = {2018}
}