Topologically Densified Distributions
Christoph Hofer, Florian Graf, Marc Niethammer, Roland Kwitt
Abstract
We study regularization in the context of small sample-size learning with over-parametrized neural networks. Specifically, we shift focus from architectural properties, such as norms on the network weights, to properties of the internal representations before a linear classifier. Specifically, we impose a topological constraint on samples drawn from the probability measure induced in that space. This provably leads to mass concentration effects around the representations of training instances, i.e., a property beneficial for generalization. By leveraging previous work to impose topological constrains in a neural network setting, we provide empirical evidence (across various vision benchmarks) to support our claim for better generalization.
BibTeX
@InProceedings{pmlr-v119-hofer20a,
title = {Topologically Densified Distributions},
author = {Hofer, Christoph and Graf, Florian and Niethammer, Marc and Kwitt, Roland},
booktitle = {Proceedings of the 37th International Conference on Machine Learning},
pages = {4304--4313},
year = {2020},
editor = {III, Hal Daumé and Singh, Aarti},
volume = {119},
series = {Proceedings of Machine Learning Research},
month = {13--18 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v119/hofer20a/hofer20a.pdf},
url = {https://proceedings.mlr.press/v119/hofer20a.html},
abstract = {We study regularization in the context of small sample-size learning with over-parametrized neural networks. Specifically, we shift focus from architectural properties, such as norms on the network weights, to properties of the internal representations before a linear classifier. Specifically, we impose a topological constraint on samples drawn from the probability measure induced in that space. This provably leads to mass concentration effects around the representations of training instances, i.e., a property beneficial for generalization. By leveraging previous work to impose topological constrains in a neural network setting, we provide empirical evidence (across various vision benchmarks) to support our claim for better generalization.}
}