Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate
Huangyu Xu, Jingqin Yang, Qianqian Xu, Jiaye Teng
Abstract
Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is Lp regularization. However, it may encounter optimization instability due to the unbounded gradients when 0<p<1. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to lp-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the L1-regularization approach while preserving test accuracy.
BibTeX
@inproceedings{
xu2026theoretical,
title={Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate},
author={Huangyu Xu and Jingqin Yang and Qianqian Xu and Jiaye Teng},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=74PNBqBKPA}
}