Oliver Astrand

Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization

The early phase of training a deep neural network has a dramatic effect on the local curvature of the loss function. For instance, using a small learning rate does not guarantee stable optimization because the optimization trajectory has a tendency to steer towards regions of the loss surface with i…