On Role and Location of Normalization before Model-based Data Augmentation in Residual Blocks for Classification Tasks

Abstract

Regularization is crucial to the success of many practical deep learning models, in particular in frequent scenarios where there are only a few to a moderate number of accessible training samples. In addition to weight decay, noise injection and dropout, regularization based on multi-branch architectures, such as Shake-Shake regularization, has been proven successful in many applications and attracted more and more attention. However, beyond model-based representation augmentation, it is unclear how Shake-Shake regularization helps to provide further improvement on classification tasks, let alone the baffling interaction between batch normalization and shaking. In this work, we present our investigation on Shake-Shake regularization. One of our findings illustrates the phenomenon that batch normalization in residual blocks is indispensable when shaking is applied to model branches, along with which we also empirically demonstrate the most effective location to place a batch normalization layer in a shaking regularized residual block. Based on these findings, we believe our work is beneficial to future studies on the research topic of refining control for model-based representation augmentation.