Global optimality of Elman-type RNNs in the mean-field regime

Andrea Agazzi; Jianfeng Lu; Sayan Mukherjee

ICML 2023poster1 citations

Global optimality of Elman-type RNNs in the mean-field regime

Andrea Agazzi, Jianfeng Lu, Sayan Mukherjee

Abstract

We analyze Elman-type recurrent neural networks (RNNs) and their training in the mean-field regime. Specifically, we show convergence of gradient descent training dynamics of the RNN to the corresponding mean-field formulation in the large width limit. We also show that the fixed points of the limiting infinite-width dynamics are globally optimal, under some assumptions on the initialization of the weights. Our results establish optimality for feature-learning with wide RNNs in the mean-field regime.

BibTeX

@inproceedings{icml2023_globaloptimality,
  title = {Global optimality of Elman-type RNNs in the mean-field regime},
  author = {Andrea Agazzi and Jianfeng Lu and Sayan Mukherjee},
  booktitle = {ICML 2023},
  year = {2023}
}