Cheng Tai

Stochastic Modified Equations and Adaptive Stochastic Gradient Algorithms

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-par…