Haoyu Sheng

Knowledge Distillation for Recurrent Neural Network Language Modeling with Trust Regularization

Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs is used. These models have large memory footpri…