Filtering Variational Objectives
Chris J Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, Yee Teh
Abstract
When used as a surrogate objective for maximum likelihood estimation in latent variable models, the evidence lower bound (ELBO) produces state-of-the-art results. Inspired by this, we consider the extension of the ELBO to a family of lower bounds defined by a particle filter's estimator of the marginal likelihood, the filtering variational objectives (FIVOs). FIVOs take the same arguments as the ELBO, but can exploit a model's sequential structure to form tighter bounds. We present results that relate the tightness of FIVO's bound to the variance of the particle filter's estimator by considering the generic case of bounds defined as log-transformed likelihood estimators. Experimentally, we show that training with FIVO results in substantial improvements over training the same model architecture with the ELBO on sequential data.
BibTeX
@inproceedings{NIPS2017_fa84632d,
author = {Maddison, Chris J and Lawson, John and Tucker, George and Heess, Nicolas and Norouzi, Mohammad and Mnih, Andriy and Doucet, Arnaud and Teh, Yee},
booktitle = {Advances in Neural Information Processing Systems},
editor = {I. Guyon and U. Von Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R. Garnett},
pages = {},
publisher = {Curran Associates, Inc.},
title = {Filtering Variational Objectives},
url = {https://proceedings.neurips.cc/paper_files/paper/2017/file/fa84632d742f2729dc32ce8cb5d49733-Paper.pdf},
volume = {30},
year = {2017}
}