2026
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
ICML 2026poster
Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar …