AIConfPaper
ICML 2019oral70 citations

Efficient Full-Matrix Adaptive Regularization

Naman Agarwal, Brian Bullins, Xinyi Chen, Elad Hazan, Karan Singh, Cyril Zhang, Yi Zhang

Source ↗PDF ↗Scholar ↗

Abstract

Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and effective. We also provide a novel theoretical analysis for adaptive regularization in

BibTeX
@InProceedings{pmlr-v97-agarwal19b,
  title = 	 {Efficient Full-Matrix Adaptive Regularization},
  author =       {Agarwal, Naman and Bullins, Brian and Chen, Xinyi and Hazan, Elad and Singh, Karan and Zhang, Cyril and Zhang, Yi},
  booktitle = 	 {Proceedings of the 36th International Conference on Machine Learning},
  pages = 	 {102--110},
  year = 	 {2019},
  editor = 	 {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
  volume = 	 {97},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {09--15 Jun},
  publisher =    {PMLR},
  pdf = 	 {http://proceedings.mlr.press/v97/agarwal19b/agarwal19b.pdf},
  url = 	 {https://proceedings.mlr.press/v97/agarwal19b.html},
  abstract = 	 {Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and effective. We also provide a novel theoretical analysis for adaptive regularization in
Efficient Full-Matrix Adaptive Regularization · ICML 2019