NeurIPS 2020poster75 citations
On the distance between two neural networks and the stability of learning
Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu
Abstract
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is here: https://github.com/jxbz/fromage.
BibTeX
@inproceedings{NEURIPS2020_f4b31bee,
author = {Bernstein, Jeremy and Vahdat, Arash and Yue, Yisong and Liu, Ming-Yu},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {21370--21381},
publisher = {Curran Associates, Inc.},
title = {On the distance between two neural networks and the stability of learning},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/f4b31bee138ff5f7b84ce1575a738f95-Paper.pdf},
volume = {33},
year = {2020}
}