Klaus Neymeyr

Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

Normalization techniques such as Batch Normalization have been applied very successfully for training deep neural networks. Yet, despite its apparent empirical benefits, the reasons behind the success of Batch Normalization are mostly hypothetical. We here aim to provide a more thorough theoretical…