Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck
Anirban Samaddar, Sandeep Madireddy, Prasanna Balaprakash, Taps Maiti, Gustavo de los Campos, Ian Fischer
Abstract
The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it provides a mechanism for learning a joint distribution of the latent variable and the sparsity, and hence it can account for the complete uncertainty in the latent space. Through a series of experiments using in-distribution and out-of-distribution learning scenarios on the MNIST, CIFAR-10, and ImageNet data, we show that the proposed approach improves accuracy and robustness compared to traditional fixed-dimensional priors, as well as other sparsity induction mechanisms for latent variable models proposed in the literature.
BibTeX
@InProceedings{pmlr-v206-samaddar23a,
title = {Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck},
author = {Samaddar, Anirban and Madireddy, Sandeep and Balaprakash, Prasanna and Maiti, Taps and de los Campos, Gustavo and Fischer, Ian},
booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics},
pages = {10207--10222},
year = {2023},
editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem},
volume = {206},
series = {Proceedings of Machine Learning Research},
month = {25--27 Apr},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v206/samaddar23a/samaddar23a.pdf},
url = {https://proceedings.mlr.press/v206/samaddar23a.html},
abstract = {The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it provides a mechanism for learning a joint distribution of the latent variable and the sparsity, and hence it can account for the complete uncertainty in the latent space. Through a series of experiments using in-distribution and out-of-distribution learning scenarios on the MNIST, CIFAR-10, and ImageNet data, we show that the proposed approach improves accuracy and robustness compared to traditional fixed-dimensional priors, as well as other sparsity induction mechanisms for latent variable models proposed in the literature.}
}