Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild
Common choices of architecture give neural networks a preference for fitting data with simple functions. This simplicity bias is known as key to their success. This paper explores the limits of this assumption. Building on recent work that showed that activation functions are the origin of the simpl…