Pierfrancesco Beneventano

Momentum Further Constrains Sharpness at the Edge of Stochastic Stability

Recent work suggests that (stochastic) gradient descent self-organizes near the instability boundary, shaping both optimization and the solutions found. Momentum and mini-batch gradients are widely used in practical deep learning optimization, but it remains unclear whether they operate in a compara…