2025
From Gradient Clipping to Normalization for Heavy Tailed SGD
AISTATS 2025poster
Recent empirical evidence indicates that many machine learning applications involve heavy-tailed gradient noise, which challenges the standard assumptions of bounded variance in stochastic optimization. Gradient clipping has emerged as a popular tool to handle this heavy-tailed noise, as it achieves…