Learning Recurrent Neural Network Language Models With Context-Sensitive Label Smoothing for Automatic Speech Recognition

Abstract

Recurrent neural network language models (RNNLMs) have become very successful in many natural language processing tasks. However, RNNLMs trained with a cross entropy loss function and hard output targets are prone to overfitting, which weakens the language models’ generalization power. In the current work, we investigate a new strategy of label smoothing in place of hard output targets to regularize RNNLM training. We propose an approach of context-sensitive candidate label smoothing that has two advantages. First, it not only helps prevent overfitted model but also distinguishes plausible words from implausible ones. Second, it helps alleviate the problems of data sparsity and unbalanced word occurrence in training data. We evaluate our proposed candidate label smoothing method on RNNLM training for two speech recognition tasks, and demonstrate its positive impacts on test set word error rate and perplexity.