AISTATS 2024poster1 citations
Implicit Regularization in Deep Tucker Factorization: Low-Rankness via Structured Sparsity
Kais Hariz, Hachem Kadri, Stéphane Ayache, Maher Moakher, Thierry Artières
Abstract
We theoretically analyze the implicit regularization of deep learning for tensor completion. We show that deep Tucker factorization trained by gradient descent induces a structured sparse regularization. This leads to a characterization of the effect of the depth of the neural network on the implicit regularization and provides a potential explanation for the bias of gradient descent towards solutions with low multilinear rank. Numerical experiments confirm our theoretical findings and give insights into the behavior of gradient descent in deep tensor factorization.
BibTeX
@InProceedings{pmlr-v238-hariz24a,
title = {Implicit Regularization in Deep {T}ucker Factorization: Low-Rankness via Structured Sparsity},
author = {Hariz, Kais and Kadri, Hachem and Ayache, St\'{e}phane and Moakher, Maher and Arti\`{e}res, Thierry},
booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics},
pages = {2359--2367},
year = {2024},
editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen},
volume = {238},
series = {Proceedings of Machine Learning Research},
month = {02--04 May},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v238/hariz24a/hariz24a.pdf},
url = {https://proceedings.mlr.press/v238/hariz24a.html},
abstract = {We theoretically analyze the implicit regularization of deep learning for tensor completion. We show that deep Tucker factorization trained by gradient descent induces a structured sparse regularization. This leads to a characterization of the effect of the depth of the neural network on the implicit regularization and provides a potential explanation for the bias of gradient descent towards solutions with low multilinear rank. Numerical experiments confirm our theoretical findings and give insights into the behavior of gradient descent in deep tensor factorization.}
}