ICML 2022spotlight5 citations
Implicit Regularization with Polynomial Growth in Deep Tensor Factorization
Kais Hariz, Hachem Kadri, Stephane Ayache, Maher Moakher, Thierry Artieres
Abstract
We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and ’shallow’ tensor factorization via linear and certain type of non-linear neural networks promotes low-rank solutions with at most quadratic growth, we show that its effect in deep tensor factorization grows polynomially with the depth of the network. This provides a remarkably faithful description of the observed experimental behaviour. Using numerical experiments, we demonstrate the benefits of this implicit regularization in yielding a more accurate estimation and better convergence properties.
BibTeX
@InProceedings{pmlr-v162-hariz22a,
title = {Implicit Regularization with Polynomial Growth in Deep Tensor Factorization},
author = {Hariz, Kais and Kadri, Hachem and Ayache, Stephane and Moakher, Maher and Artieres, Thierry},
booktitle = {Proceedings of the 39th International Conference on Machine Learning},
pages = {8484--8501},
year = {2022},
editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
volume = {162},
series = {Proceedings of Machine Learning Research},
month = {17--23 Jul},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v162/hariz22a/hariz22a.pdf},
url = {https://proceedings.mlr.press/v162/hariz22a.html},
abstract = {We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and ’shallow’ tensor factorization via linear and certain type of non-linear neural networks promotes low-rank solutions with at most quadratic growth, we show that its effect in deep tensor factorization grows polynomially with the depth of the network. This provides a remarkably faithful description of the observed experimental behaviour. Using numerical experiments, we demonstrate the benefits of this implicit regularization in yielding a more accurate estimation and better convergence properties.}
}