Damien MARTINS GOMES

AdaFisher: Adaptive Second Order Optimization via Fisher Information

First-order optimization methods are currently the mainstream in training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by employing the diagonal matrix preconditioning of the stochastic gradient during the training. Despite their widespread, second-orde…