Dwarak Govind Parthiban

On the Softmax Bottleneck of Recurrent Language Models

Recent research has pointed to a limitation of word-level neural language models with softmax outputs. This limitation, known as the softmax bottleneck refers to the inability of these models to produce high-rank log probability (log P) matrices. Various solutions have been proposed to break this bo…