FOSI: Hybrid First and Second Order Optimization
Hadar Sivan, Moshe Gabel, Assaf Schuster
Abstract
Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions. We present FOSI, a novel meta-algorithm that improves the performance of any base first-order optimizer by efficiently incorporating second-order information during the optimization process. In each iteration, FOSI implicitly splits the function into two quadratic functions defined on orthogonal subspaces, then uses a second-order method to minimize the first, and the base optimizer to minimize the other. We formally analyze FOSI's convergence and the conditions under which it improves a base optimizer. Our empirical evaluation demonstrates that FOSI improves the convergence rate and optimization time of first-order methods such as Heavy-Ball and Adam, and outperforms second-order methods (K-FAC and L-BFGS).
BibTeX
@inproceedings{
sivan2024fosi,
title={{FOSI}: Hybrid First and Second Order Optimization},
author={Hadar Sivan and Moshe Gabel and Assaf Schuster},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=NvbeD9Ttkx}
}