2020
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
NeurIPS 2020spotlight
Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g.~Adam) and accelerated schemes (e.g.~stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize…