Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, Andrew Gordon Wilson
Abstract
The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We prove non-asymptotic convergence theory of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.
BibTeX
@inproceedings{
Zhang2020Cyclical,
title={Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning},
author={Ruqi Zhang and Chunyuan Li and Jianyi Zhang and Changyou Chen and Andrew Gordon Wilson},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=rkeS1RVtPS}
}