Kuangyu Ding

Memory-Efficient 4-bit Preconditioned Stochastic Optimization

Preconditioned stochastic optimization algorithms, exemplified by Shampoo, outperform first-order optimizers by offering theoretical convergence benefits and practical gains in large-scale neural network training. However, they incur substantial memory overhead due to the storage demands of non-diag…