Ryuhei Kikuchi

Why is parameter averaging beneficial in SGD? An objective smoothing perspective

It is often observed that stochastic gradient descent (SGD) and its variants implicitly select a solution with good generalization performance; such implicit bias is often characterized in terms of the sharpness of the minima. Kleinberg et al. (2018) connected this bias with the smoothing effect of…