Reza Gheissari — accepted AI-conference papers

2024

High-dimensional SGD aligns with emerging outlier eigenspaces

Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, Aukosh Jagannath

ICLR 2024spotlight

We rigorously study the joint evolution of training dynamics via stochastic gradient descent (SGD) and the spectra of empirical Hessian and gradient matrices. We prove that in two canonical classification tasks for multi-class high-dimensional mixtures and either 1 or 2-layer neural networks, the SG…

Cited by 11 Source ↗PDF ↗Scholar ↗

2022

High-dimensional limit theorems for SGD: Effective dynamics and critical scaling

Gerard Ben Arous, Reza Gheissari, Aukosh Jagannath

NeurIPS 2022accept

We study the scaling limits of stochastic gradient descent (SGD) with constant step-size in the high-dimensional regime. We prove limit theorems for the trajectories of summary statistics (i.e., finite-dimensional functions) of SGD as the dimension goes to infinity. Our approach allows one to choose…

Cited by 87 Source ↗PDF ↗Scholar ↗