Representation Costs of Linear Neural Networks: Analysis and Design
Zhen Dai, Mina Karzand, Nathan Srebro
Abstract
For different parameterizations (mappings from parameters to predictors), we study the regularization cost in predictor space induced by $l_2$ regularization on the parameters (weights). We focus on linear neural networks as parameterizations of linear predictors. We identify the representation cost of certain sparse linear ConvNets and residual networks. In order to get a better understanding of how the architecture and parameterization affect the representation cost, we also study the reverse problem, identifying which regularizers on linear predictors (e.g., $l_p$ norms, group norms, the $k$-support-norm, elastic net) can be the representation cost induced by simple $l_2$ regularization, and designing the parameterizations that do so.
BibTeX
@inproceedings{
dai2021representation,
title={Representation Costs of Linear Neural Networks: Analysis and Design},
author={Zhen Dai and Mina Karzand and Nathan Srebro},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=pSNs0PKx0Mw}
}