The asymptotic spectrum of the Hessian of DNN throughout training

Arthur Jacot; Franck Gabriel; Clement Hongler

ICLR 2020poster33 citations

The asymptotic spectrum of the Hessian of DNN throughout training

Arthur Jacot, Franck Gabriel, Clement Hongler

Abstract

The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs: we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training.

theory of deep learningloss surfacetrainingfisher information matrix

BibTeX

@inproceedings{
Jacot2020The,
title={The asymptotic spectrum of the Hessian of DNN throughout training},
author={Arthur Jacot and Franck Gabriel and Clement Hongler},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=SkgscaNYPS}
}