Continuous visual speech recognition for audio speech enhancement
Eric Benhaim, Hichem Sahbi, Guillaume Vitte
Abstract
We introduce in this paper a novel non-blind speech enhancement procedure based on visual speech recognition (VSR). The latter is based on a generative process that analyzes sequences of talking faces and classifies them into visual speech units known as visemes. We use an effective graphical model able to segment and label a given sequence of talking faces into a sequence of visemes. Our model captures unary potential as well as pairwise interaction; the former models visual appearance of speech units while the latter models their interactions using boundary and visual language model activations. Experiments conducted on a standard challenging dataset, show that when feeding the results of VSR to the speech enhancement procedure, it clearly outperforms baseline blind methods as well as related work.
BibTeX
@inproceedings{icassp2015_continuousvisual,
title = {Continuous visual speech recognition for audio speech enhancement},
author = {Eric Benhaim and Hichem Sahbi and Guillaume Vitte},
booktitle = {ICASSP 2015},
year = {2015}
}