Zach Eaton-Rosen — accepted AI-conference papers

2021

Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence

Antoine Labatie, Dominic Masters, Zach Eaton-Rosen, Carlo Luschi

NeurIPS 2021poster

We investigate the reasons for the performance degradation incurred with batch-independent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appearance of failure modes in the neural network's pre-activations: (i) layer normaliz…

Cited by 25 Source ↗PDF ↗Scholar ↗

2020

Improving Neural Network Training in Low Dimensional Random Bases

Frithjof Gressmann, Zach Eaton-Rosen, Carlo Luschi

NeurIPS 2020poster

Stochastic Gradient Descent (SGD) has proven to be remarkably effective in optimizing deep neural networks that employ ever-larger numbers of parameters. Yet, improving the efficiency of large-scale optimization remains a vital and highly active area of research. Recent work has shown that deep neur…

Cited by 29 Source ↗PDF ↗code ↗Scholar ↗