2022
Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentum
ICLR 2022poster
Most convergence guarantees for stochastic gradient descent with momentum (SGDm) rely on iid sampling. Yet, SGDm is often used outside this regime, in settings with temporally correlated input samples such as continual learning and reinforcement learning. Existing work has shown that SGDm with a de…