End-to-End Training of Time Domain Audio Separation and Recognition
Thilo von Neumann, Keisuke Kinoshita, Lukas Drude, Christoph Böddeker, Marc Delcroix, Tomohiro Nakatani, Reinhold Haeb-Umbach
Abstract
The rising interest in single-channel multi-speaker speech separation sparked development of End-to-End (E2E) approaches to multi-speaker speech recognition. However, up until now, state-of-the-art neural network-based time domain source separation has not yet been combined with E2E speech recognition. We here demonstrate how to combine a separation module based on a Convolutional Time domain Audio Separation Network (Conv-TasNet) with an E2E speech recognizer and how to train such a model jointly by distributing it over multiple GPUs or by approximating truncated back-propagation for the convolutional front-end. To put this work into perspective and illustrate the complexity of the design space, we provide a compact overview of single-channel multi-speaker recognition systems. Our experiments show a word error rate of 11.0% on WSJ0-2mix and indicate that our joint time domain model can yield substantial improvements over cascade DNN-HMM and monolithic E2E frequency domain systems proposed so far.
BibTeX
@inproceedings{icassp2020_endtoendtraining,
title = {End-to-End Training of Time Domain Audio Separation and Recognition},
author = {Thilo von Neumann and Keisuke Kinoshita and Lukas Drude and Christoph Böddeker and Marc Delcroix and Tomohiro Nakatani and Reinhold Haeb-Umbach},
booktitle = {ICASSP 2020},
year = {2020}
}