DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
Arash Vahdat, William Macready, Zhengbing Bian, Amir Khoshaman, Evgeny Andriyash
Abstract
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).
BibTeX
@InProceedings{pmlr-v80-vahdat18a,
title = {{DVAE}++: Discrete Variational Autoencoders with Overlapping Transformations},
author = {Vahdat, Arash and Macready, William and Bian, Zhengbing and Khoshaman, Amir and Andriyash, Evgeny},
booktitle = {Proceedings of the 35th International Conference on Machine Learning},
pages = {5035--5044},
year = {2018},
editor = {Dy, Jennifer and Krause, Andreas},
volume = {80},
series = {Proceedings of Machine Learning Research},
month = {10--15 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v80/vahdat18a/vahdat18a.pdf},
url = {https://proceedings.mlr.press/v80/vahdat18a.html},
abstract = {Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).}
}