Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, William W. Cohen
Abstract
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.
BibTeX
@inproceedings{
yang2018breaking,
title={Breaking the Softmax Bottleneck: A High-Rank {RNN} Language Model},
author={Zhilin Yang and Zihang Dai and Ruslan Salakhutdinov and William W. Cohen},
booktitle={International Conference on Learning Representations},
year={2018},
url={https://openreview.net/forum?id=HkwZSG-CZ},
}