Lookahead Converges to Stationary Points of Smooth Non-convex Functions

Jianyu Wang; Vinayak Tantia; Nicolas Ballas; Michael G. Rabbat

ICASSP 2020accepted0 citations

Lookahead Converges to Stationary Points of Smooth Non-convex Functions

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael G. Rabbat

Abstract

The Lookahead optimizer [Zhang et al., 2019] was recently proposed and demonstrated to improve performance of stochastic first-order methods for training deep neural networks. Lookahead can be viewed as a two time-scale algorithm, where the fast dynamics (inner optimizer) determine a search direction and the slow dynamics (outer optimizer) perform updates by moving along this direction. We prove that, with appropriate choice of step-sizes, Lookahead converges to a stationary point of smooth non-convex functions. Although Lookahead is described and implemented as a serial algorithm, our analysis is based on viewing Lookahead as a multi-agent optimization method with two agents communicating periodically.

BibTeX

@inproceedings{icassp2020_lookaheadconverg,
  title = {Lookahead Converges to Stationary Points of Smooth Non-convex Functions},
  author = {Jianyu Wang and Vinayak Tantia and Nicolas Ballas and Michael G. Rabbat},
  booktitle = {ICASSP 2020},
  year = {2020}
}