An Adaptive Empirical Bayesian Method for Sparse Deep Learning
Wei Deng, Xiao Zhang, Faming Liang, Guang Lin
Abstract
We propose a novel adaptive empirical Bayesian (AEB) method for sparse deep learning, where the sparsity is ensured via a class of self-adaptive spike-and-slab priors. The proposed method works by alternatively sampling from an adaptive hierarchical posterior distribution using stochastic gradient Markov Chain Monte Carlo (MCMC) and smoothly optimizing the hyperparameters using stochastic approximation (SA). The convergence of the proposed method to the asymptotically correct distribution is established under mild conditions. Empirical applications of the proposed method lead to the state-of-the-art performance on MNIST and Fashion MNIST with shallow convolutional neural networks (CNN) and the state-of-the-art compression performance on CIFAR10 with Residual Networks. The proposed method also improves resistance to adversarial attacks.
BibTeX
@inproceedings{NEURIPS2019_2d71b2ae,
author = {Deng, Wei and Zhang, Xiao and Liang, Faming and Lin, Guang},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {},
publisher = {Curran Associates, Inc.},
title = {An Adaptive Empirical Bayesian Method for Sparse Deep Learning},
url = {https://proceedings.neurips.cc/paper_files/paper/2019/file/2d71b2ae158c7c5912cc0bbde2bb9d95-Paper.pdf},
volume = {32},
year = {2019}
}