NeurIPS 2020oral66 citations
Escaping the Gravitational Pull of Softmax
Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvari, Dale Schuurmans
Abstract
The softmax is the standard transformation used in machine learning to map real-valued vectors to categorical distributions. Unfortunately, this transform poses serious drawbacks for gradient descent (ascent) optimization. We reveal this difficulty by establishing two negative results: (1) optimizing any expectation with respect to the softmax must exhibit sensitivity to parameter initialization (
BibTeX
@inproceedings{NEURIPS2020_f1cf2a08,
author = {Mei, Jincheng and Xiao, Chenjun and Dai, Bo and Li, Lihong and Szepesvari, Csaba and Schuurmans, Dale},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {21130--21140},
publisher = {Curran Associates, Inc.},
title = {Escaping the Gravitational Pull of Softmax},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/f1cf2a082126bf02de0b307778ce73a7-Paper.pdf},
volume = {33},
year = {2020}
}