Timofey Belinsky

Softsignum: Smooth Your Signum For Better Heterogeneity Handling

Sign-based optimization methods, such as SignSGD and Signum, have become essential for modern Deep Learning due to their 1) high performance 2) low memory footprint and 3) communication efficiency. Despite their success, these methods suffer from distinct limitations in the terminal phase of trainin…