2022
Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability
NeurIPS 2022accept
Recent findings demonstrate that modern neural networks trained by full-batch gradient descent typically enter a regime called Edge of Stability (EOS). In this regime, the sharpness, i.e., the maximum Hessian eigenvalue, first increases to the value 2/(step size) (the progressive sharpening phase) a…