Zongwei Huo

IO-Adam: Rethinking Memory-Efficient Adaptive Optimizers from Gradient Computation

Adaptive Moment Estimation (Adam) is one of the most popular and often the default stochastic optimizers for deep neural network training. Using first- and second-moment estimation, Adam provides adaptive learning rates for each parameter, significantly outperforming Stochastic Gradient Descent (SGD…