Acceleration of SVRG and Katyusha X by Inexact Preconditioning

Yanli Liu; Fei Feng; Wotao Yin

ICML 2019oral11 citations

Acceleration of SVRG and Katyusha X by Inexact Preconditioning

Abstract

Empirical risk minimization is an important class of optimization problems with many popular machine learning applications, and stochastic variance reduction methods are popular choices for solving them. Among these methods, SVRG and Katyusha X (a Nesterov accelerated SVRG) achieve fast convergence without substantial memory requirement. In this paper, we propose to accelerate these two algorithms by

BibTeX

@InProceedings{pmlr-v97-liu19a,
  title = 	 {Acceleration of {SVRG} and {K}atyusha X by Inexact Preconditioning},
  author =       {Liu, Yanli and Feng, Fei and Yin, Wotao},
  booktitle = 	 {Proceedings of the 36th International Conference on Machine Learning},
  pages = 	 {4003--4012},
  year = 	 {2019},
  editor = 	 {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
  volume = 	 {97},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {09--15 Jun},
  publisher =    {PMLR},
  pdf = 	 {http://proceedings.mlr.press/v97/liu19a/liu19a.pdf},
  url = 	 {https://proceedings.mlr.press/v97/liu19a.html},
  abstract = 	 {Empirical risk minimization is an important class of optimization problems with many popular machine learning applications, and stochastic variance reduction methods are popular choices for solving them. Among these methods, SVRG and Katyusha X (a Nesterov accelerated SVRG) achieve fast convergence without substantial memory requirement. In this paper, we propose to accelerate these two algorithms by