Stijn De Baerdemacker

Almost Bayesian: Dynamics of SGD Through Singular Learning Theory

The nature of the relationship between Bayesian sampling and stochastic gradient descent in neural networks has been a long-standing open question in the theory of deep learning. We shed light on this question by modeling the long runtime behaviour of SGD as diffusion on porous media. Using singular…