Fenlong Jiang

SKFAC: Training Neural Networks With Faster Kronecker-Factored Approximate Curvature

The bottleneck of computation burden limits the widespread use of the 2nd order optimization algorithms for training deep neural networks. In this paper, we present a computationally efficient approximation for natural gradient descent, named Swift Kronecker-Factored Approximate Curvature (SKFAC), w…