Armand Mihai Nicolicioiu

Neural Redshift: Random Networks are not Random Functions

Our understanding of the generalization capabilities of neural networks NNs is still incomplete. Prevailing explanations are based on implicit biases of gradient descent GD but they cannot account for the capabilities of models from gradientfree methods nor the simplicity bias recently observed in u…