ICML 2021spotlight38 citations
Dropout: Explicit Forms and Capacity Control
Raman Arora, Peter Bartlett, Poorya Mianjy, Nathan Srebro
Abstract
We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a distribution-dependent regularizer that equals the weighted trace-norm of the product of the factors. In deep learning, we show that the distribution-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks.
BibTeX
@InProceedings{pmlr-v139-arora21a,
title = {Dropout: Explicit Forms and Capacity Control},
author = {Arora, Raman and Bartlett, Peter and Mianjy, Poorya and Srebro, Nathan},
booktitle = {Proceedings of the 38th International Conference on Machine Learning},
pages = {351--361},
year = {2021},
editor = {Meila, Marina and Zhang, Tong},
volume = {139},
series = {Proceedings of Machine Learning Research},
month = {18--24 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v139/arora21a/arora21a.pdf},
url = {https://proceedings.mlr.press/v139/arora21a.html},
abstract = {We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a distribution-dependent regularizer that equals the weighted trace-norm of the product of the factors. In deep learning, we show that the distribution-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks.}
}