On the Benefits of Weight Normalization for Overparameterized Matrix Sensing
Yudong Wei, Liang Zhang, Bingcong Li, Niao He
Abstract
While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian optimization achieves linear convergence, yielding an $\textit{exponential}$ speedup over standard methods that do not use WN. Our analysis further demonstrates that both iteration and sample complexity improve polynomially as the level of overparameterization increases. To the best of our knowledge, this work provides the first characterization of how WN leverages overparameterization for faster convergence in matrix sensing.
BibTeX
@inproceedings{
wei2026on,
title={On the Benefits of Weight Normalization for Overparameterized Matrix Sensing},
author={Yudong Wei and Liang Zhang and Bingcong Li and Niao He},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=JIMM5YLShy}
}