Application of Progressive Neural Networks for Multi-Stream Wfst Combination in One-Pass Decoding

Abstract

Many state-of-the-art automatic speech recognition (ASR) systems adopt system combination techniques to improve recognition performance. In this paper, we investigate the possibility of transferring knowledge between models for different noisy speech domains and integrating these models via system combination. The first contribution of our work is the use of progressive neural networks for modeling the acoustic features of noisy speech. We train progressive neural networks on subdivided noisy data to achieve knowledge transfer between different noise conditions. Our second contribution is an improved multi-stream WFST framework that combines the output of the progressive networks at longer timescales (e.g., word hypotheses). The score fusion is performed by a trained LSTM at the word boundary on the decoding lattice. By adopting both knowledge transfer and system combination techniques, we achieve improved performance compared with independently trained deep neural networks.