NeurIPS 2020poster89 citations
Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology
Quynh N Nguyen, Marco Mondelli
Abstract
Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with N (N being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width N following the input layer suffices to ensure a similar guarantee. In particular, all the remaining layers are allowed to have constant widths, and form a pyramidal topology. We show an application of our result to the widely used Xavier's initialization and obtain an over-parameterization requirement for the single wide layer of order N^2.
BibTeX
@inproceedings{NEURIPS2020_8abfe8ac,
author = {Nguyen, Quynh N and Mondelli, Marco},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {11961--11972},
publisher = {Curran Associates, Inc.},
title = {Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/8abfe8ac9ec214d68541fcb888c0b4c3-Paper.pdf},
volume = {33},
year = {2020}
}