Ningxi Ni

THOR, Trace-based Hardware-driven Layer-Oriented Natural Gradient Descent Computation

It is well-known that second-order optimizer can accelerate the training of deep neural networks, however, the huge computation cost of second-order optimization makes it impractical to apply in real practice. In order to reduce the cost, many methods have been proposed to approximate a second-order…