Compressing Transformer-Based ASR Model by Task-Driven Loss and Attention-Based Multi-Level Feature Distillation

Yongjie Lv; Longbiao Wang; Meng Ge; Sheng Li; Chenchen Ding; Lixin Pan; Yuguang Wang; Jianwu Dang

ICASSP 2022accepted0 citations

Compressing Transformer-Based ASR Model by Task-Driven Loss and Attention-Based Multi-Level Feature Distillation

Yongjie Lv, Longbiao Wang, Meng Ge, Sheng Li, Chenchen Ding, Lixin Pan, Yuguang Wang, Jianwu Dang

Abstract

The current popular knowledge distillation (KD) methods effectively compress the transformer-based end-to-end speech recognition model. However, existing methods fail to utilize complete information of the teacher model, and they distill only a limited number of blocks of the teacher model. In this study, we first integrate a task-driven loss function into the decoder’s intermediate blocks to generate task-related feature representations. Then, we propose an attention-based multi-level feature distillation to automatically learn the feature representation summarized by all blocks of the teacher model. Under the 1.1M parameters model, the experimental results on the Wall Street Journal dataset reveal that our approach achieves a 12.1% WER reduction compared with the baseline system.

BibTeX

@inproceedings{icassp2022_compressingtrans,
  title = {Compressing Transformer-Based ASR Model by Task-Driven Loss and Attention-Based Multi-Level Feature Distillation},
  author = {Yongjie Lv and Longbiao Wang and Meng Ge and Sheng Li and Chenchen Ding and Lixin Pan and Yuguang Wang and Jianwu Dang and Kiyoshi Honda},
  booktitle = {ICASSP 2022},
  year = {2022}
}