NeurIPS 2022accept6 citations
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
Libin Zhu, Chaoyue Liu, Misha Belkin
Abstract
In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their ``width'' approaches infinity. The width of these general networks is characterized by the minimum in-degree of their neurons, except for the input and first layers. Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the Neural Tangent Kernel for standard architectures.
wide neural networksdirected acyclic graphtransition to linearityneural tangent kernelover-parameterization
BibTeX
@inproceedings{
zhu2022transition,
title={Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture},
author={Libin Zhu and Chaoyue Liu and Misha Belkin},
booktitle={Advances in Neural Information Processing Systems},
editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
year={2022},
url={https://openreview.net/forum?id=Tean8bBjlbB}
}