Lianhong Ding

AdaR: An Adaptive Gradient Method with Cyclical Restarting of Moment Estimations

Adaptive gradient methods, primarily based on Adam, are prevalent in training neural networks, adjusting step sizes via exponentially decaying averages of gradients and squared gradients. Adam assigns small weights to distant gradients, termed long-tail gradients in this paper. However, these gradie