Stochastic Adaptive Quasi-Newton Methods for Minimizing Expected Values
Chaoxu Zhou, Wenbo Gao, Donald Goldfarb
Abstract
We propose a novel class of stochastic, adaptive methods for minimizing self-concordant functions which can be expressed as an expected value. These methods generate an estimate of the true objective function by taking the empirical mean over a sample drawn at each step, making the problem tractable. The use of adaptive step sizes eliminates the need for the user to supply a step size. Methods in this class include extensions of gradient descent (GD) and BFGS. We show that, given a suitable amount of sampling, the stochastic adaptive GD attains linear convergence in expectation, and with further sampling, the stochastic adaptive BFGS attains R-superlinear convergence. We present experiments showing that these methods compare favorably to SGD.
BibTeX
@InProceedings{pmlr-v70-zhou17a,
title = {Stochastic Adaptive Quasi-{N}ewton Methods for Minimizing Expected Values},
author = {Chaoxu Zhou and Wenbo Gao and Donald Goldfarb},
booktitle = {Proceedings of the 34th International Conference on Machine Learning},
pages = {4150--4159},
year = {2017},
editor = {Precup, Doina and Teh, Yee Whye},
volume = {70},
series = {Proceedings of Machine Learning Research},
month = {06--11 Aug},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v70/zhou17a/zhou17a.pdf},
url = {https://proceedings.mlr.press/v70/zhou17a.html},
abstract = {We propose a novel class of stochastic, adaptive methods for minimizing self-concordant functions which can be expressed as an expected value. These methods generate an estimate of the true objective function by taking the empirical mean over a sample drawn at each step, making the problem tractable. The use of adaptive step sizes eliminates the need for the user to supply a step size. Methods in this class include extensions of gradient descent (GD) and BFGS. We show that, given a suitable amount of sampling, the stochastic adaptive GD attains linear convergence in expectation, and with further sampling, the stochastic adaptive BFGS attains R-superlinear convergence. We present experiments showing that these methods compare favorably to SGD.}
}