ICLR 2024poster24 citations
Quadratic models for understanding catapult dynamics of neural networks
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin
Abstract
While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we show that recently proposed Neural Quadratic Models can exhibit the "catapult phase" Lewkowycz et al. (2020) that arises when training such models with large learning rates. We then empirically show that the behaviour of quadratic models parallels that of neural networks in generalization, especially in the catapult phase regime. Our analysis further demonstrates that quadratic models are an effective tool for analysis of neural networks.
quadratic modelswide neural networkscatapult phaseoptimization dynamics
BibTeX
@inproceedings{
zhu2024quadratic,
title={Quadratic models for understanding catapult dynamics of neural networks},
author={Libin Zhu and Chaoyue Liu and Adityanarayanan Radhakrishnan and Mikhail Belkin},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PvJnX3dwsD}
}