2018
Exact natural gradient in deep linear networks and its application to the nonlinear case
NeurIPS 2018poster
Stochastic gradient descent (SGD) remains the method of choice for deep learning, despite the limitations arising for ill-behaved objective functions. In cases where it could be estimated, the natural gradient has proven very effective at mitigating the catastrophic effects of pathological curvature…