Sam Smith

Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks. We show that this key benefit arises because, at initialization, batch normalization downsc…