Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support
Tim Reichelt, Luke Ong, Tom Rainforth
Abstract
The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.
BibTeX
@InProceedings{pmlr-v238-reichelt24a,
title = {Beyond {B}ayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support},
author = {Reichelt, Tim and Ong, Luke and Rainforth, Tom},
booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics},
pages = {829--837},
year = {2024},
editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen},
volume = {238},
series = {Proceedings of Machine Learning Research},
month = {02--04 May},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v238/reichelt24a/reichelt24a.pdf},
url = {https://proceedings.mlr.press/v238/reichelt24a.html},
abstract = {The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.}
}