2026
AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
ICML 2026poster
Loss spikes remain a persistent obstacle in large-scale language model pretraining. While previous research has attempted to identify the root cause of loss spikes by investigating individual factors, we observe that, in practice, such spikes are typically triggered by the confluence of heterogeneou…