Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, David Duvenaud
Abstract
Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables, based on gradients of a learned function. These estimators can be jointly trained with model parameters or policies, and are applicable in both discrete and continuous settings. We give unbiased, adaptive analogs of state-of-the-art reinforcement learning methods such as advantage actor-critic. We also demonstrate this framework for training discrete latent-variable models.
BibTeX
@inproceedings{
grathwohl2018backpropagation,
title={Backpropagation through the Void: Optimizing control variates for black-box gradient estimation},
author={Will Grathwohl and Dami Choi and Yuhuai Wu and Geoff Roeder and David Duvenaud},
booktitle={International Conference on Learning Representations},
year={2018},
url={https://openreview.net/forum?id=SyzKd1bCW},
}