Activation-Free Backbones for Image Recognition: Polynomial Alternatives for Spatial and Channel Mixing
Jeffrey Wang, Jonathan Gregory, Grigorios Chrysos
Abstract
Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures. Our code is available at https://anonymous.4open.science/r/PolyNeXt-E424
BibTeX
@inproceedings{
wang2026activationfree,
title={Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models},
author={Jeffrey Wang and Jonathan Gregory and Grigorios Chrysos},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=FR1XRg47fL}
}