Zhenghai Huang — accepted AI-conference papers

2021

A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

Kaixin Gao, Xiaolei Liu, Zhenghai Huang, Min Wang, Zidong Wang, Dachuan Xu +1

AAAI 2021technical

Second-order optimization methods have the ability to accelerate convergence by modifying the gradient through the curvature matrix. There have been many attempts to use second-order optimization methods for training deep neural networks. In this work, inspired by diagonal approximations and factore…

Cited by 13 Source ↗PDF ↗Scholar ↗

2021

THOR, Trace-based Hardware-driven Layer-Oriented Natural Gradient Descent Computation

Mengyun Chen, Kaixin Gao, Xiaolei Liu, Zidong Wang, Ningxi Ni, Qian Zhang +8

AAAI 2021technical

It is well-known that second-order optimizer can accelerate the training of deep neural networks, however, the huge computation cost of second-order optimization makes it impractical to apply in real practice. In order to reduce the cost, many methods have been proposed to approximate a second-order…

Cited by 9 Source ↗PDF ↗Scholar ↗