NeurIPS 2019poster141 citations
On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
Abstract
The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.
BibTeX
@inproceedings{NEURIPS2019_84d2004b,
author = {Defazio, Aaron and Bottou, Leon},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {},
publisher = {Curran Associates, Inc.},
title = {On the Ineffectiveness of Variance Reduced Optimization for Deep Learning},
url = {https://proceedings.neurips.cc/paper_files/paper/2019/file/84d2004bf28a2095230e8e14993d398d-Paper.pdf},
volume = {32},
year = {2019}
}