2026
Activation Function Design Sustains Plasticity in Continual Learning
ICLR 2026poster
In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differences often shrink once model size and optimization are tuned. In continual learning, however, the picture is different: beyond catastrophic forgetting, models c…