2026
Softsignum: Smooth Your Signum For Better Heterogeneity Handling
ICML 2026poster
Sign-based optimization methods, such as SignSGD and Signum, have become essential for modern Deep Learning due to their 1) high performance 2) low memory footprint and 3) communication efficiency. Despite their success, these methods suffer from distinct limitations in the terminal phase of trainin…