Tomoshi Iiyama

ADOPT: Modified Adam Can Converge with Any $\beta_2$ with the Optimal Rate

Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., $\beta_2$, in a problem-dependent manner. There have been many attempts to fix the non-convergence (e.g., AMSGrad), but they re…