Tsung-Hsien Lee

Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner

The recent success of Shampoo in the AlgoPerf contest has sparked renewed interest in Kronecker-factorization-based optimization algorithms for training neural networks. Despite its success, Shampoo relies heavily on several heuristics such as learning rate grafting and stale preconditioning to achi…