Learning Shallow Neural Networks via Provable Gradient Descent with Random Initialization

Shuhao Xia; Yuanming Shi

ICASSP 2019accepted0 citations

Learning Shallow Neural Networks via Provable Gradient Descent with Random Initialization

Abstract

This paper presents the provable gradient descent algorithm with random initialization for learning a two-layer neural network with quadratic activation functions. Specifically, we focus on the under-parameterized regime where the number of hidden units is smaller than the dimension of the inputs. We reveal that the randomly initialized gradient descent for the nonconvex neural network training problem is able to enter a local region that enjoys strong convexity and strong smoothness within a few iterations, and then provably converges to a globally optimal model at a linear rate.

BibTeX

@inproceedings{icassp2019_learningshallown,
  title = {Learning Shallow Neural Networks via Provable Gradient Descent with Random Initialization},
  author = {Shuhao Xia and Yuanming Shi},
  booktitle = {ICASSP 2019},
  year = {2019}
}