Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
Liwei Wang, Alexander Schwing, Svetlana Lazebnik
Abstract
This paper explores image caption generation using conditional variational auto-encoders (CVAEs). Standard CVAEs with a fixed Gaussian prior yield descriptions with too little variability. Instead, we propose two models that explicitly structure the latent space around K components corresponding to different types of image content, and combine components to create priors for images that contain multiple types of content simultaneously (e.g., several kinds of objects). Our first model uses a Gaussian Mixture model (GMM) prior, while the second one defines a novel Additive Gaussian (AG) prior that linearly combines component means. We show that both models produce captions that are more diverse and more accurate than a strong LSTM baseline or a “vanilla” CVAE with a fixed Gaussian prior, with AG-CVAE showing particular promise.
BibTeX
@inproceedings{NIPS2017_4b21cf96,
author = {Wang, Liwei and Schwing, Alexander and Lazebnik, Svetlana},
booktitle = {Advances in Neural Information Processing Systems},
editor = {I. Guyon and U. Von Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R. Garnett},
pages = {},
publisher = {Curran Associates, Inc.},
title = {Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space},
url = {https://proceedings.neurips.cc/paper_files/paper/2017/file/4b21cf96d4cf612f239a6c322b10c8fe-Paper.pdf},
volume = {30},
year = {2017}
}