2021
Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
NeurIPS 2021poster
We investigate the reasons for the performance degradation incurred with batch-independent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appearance of failure modes in the neural network's pre-activations: (i) layer normaliz…