Yuta Nishikawa

Keep Decoding Parallel With Effective Knowledge Distillation From Language Models To End-To-End Speech Recognisers

This study presents a novel approach for knowledge distillation (KD) from a BERT teacher model to an automatic speech recognition (ASR) model using intermediate layers. To distil the teacher’s knowledge, we use an attention decoder that learns from BERT’s token probabilities. Our method shows that l…