Geonhui Yoo

Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More

When training deep neural networks with gradient descent, sharpness often increases---a phenomenon known as *progressive sharpening*---before saturating at the *edge of stability*. Although commonly observed in practice, the underlying mechanisms behind progressive sharpening remain poorly understoo…