ICML 2018oral509 citations
Essentially No Barriers in Neural Network Energy Landscape
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred Hamprecht
Abstract
Training neural networks involves finding minima of a high-dimensional non-convex loss function. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that minima are perhaps best seen as points on a single connected manifold of low loss, rather than as the bottoms of distinct valleys.
BibTeX
@InProceedings{pmlr-v80-draxler18a,
title = {Essentially No Barriers in Neural Network Energy Landscape},
author = {Draxler, Felix and Veschgini, Kambis and Salmhofer, Manfred and Hamprecht, Fred},
booktitle = {Proceedings of the 35th International Conference on Machine Learning},
pages = {1309--1318},
year = {2018},
editor = {Dy, Jennifer and Krause, Andreas},
volume = {80},
series = {Proceedings of Machine Learning Research},
month = {10--15 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v80/draxler18a/draxler18a.pdf},
url = {https://proceedings.mlr.press/v80/draxler18a.html},
abstract = {Training neural networks involves finding minima of a high-dimensional non-convex loss function. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that minima are perhaps best seen as points on a single connected manifold of low loss, rather than as the bottoms of distinct valleys.}
}