NeurIPS 2020poster190 citations
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
Abstract
Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work considers this question, examines the accuracy of existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian.
BibTeX
@inproceedings{NEURIPS2020_d1ff1ec8,
author = {Singh, Sidak Pal and Alistarh, Dan},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {18098--18109},
publisher = {Curran Associates, Inc.},
title = {WoodFisher: Efficient Second-Order Approximation for Neural Network Compression},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/d1ff1ec86b62cd5f3903ff19c3a326b2-Paper.pdf},
volume = {33},
year = {2020}
}