NeurIPS 2019poster19 citations
The Implicit Bias of AdaGrad on Separable Data
Abstract
We study the implicit bias of AdaGrad on separable linear classification problems. We show that AdaGrad converges to a direction that can be characterized as the solution of a quadratic optimization problem with the same feasible set as the hard SVM problem. We also give a discussion about how different choices of the hyperparameters of AdaGrad may impact this direction. This provides a deeper understanding of why adaptive methods do not seem to have the generalization ability as good as gradient descent does in practice.
BibTeX
@inproceedings{NEURIPS2019_3335881e,
author = {Qian, Qian and Qian, Xiaoyuan},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {},
publisher = {Curran Associates, Inc.},
title = {The Implicit Bias of AdaGrad on Separable Data},
url = {https://proceedings.neurips.cc/paper_files/paper/2019/file/3335881e06d4d23091389226225e17c7-Paper.pdf},
volume = {32},
year = {2019}
}