Jonathan Gregory

Activation-Free Backbones for Image Recognition: Polynomial Alternatives for Spatial and Channel Mixing

Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadama…