Residual Loss Prediction: Reinforcement Learning With No Incremental Feedback
Hal Daumé III, John Langford, Amr Sharaf
Abstract
We consider reinforcement learning and bandit structured prediction problems with very sparse loss feedback: only at the end of an episode. We introduce a novel algorithm, RESIDUAL LOSS PREDICTION (RESLOPE), that solves such problems by automatically learning an internal representation of a denser reward function. RESLOPE operates as a reduction to contextual bandits, using its learned loss representation to solve the credit assignment problem, and a contextual bandit oracle to trade-off exploration and exploitation. RESLOPE enjoys a no-regret reduction-style theoretical guarantee and outperforms state of the art reinforcement learning algorithms in both MDP environments and bandit structured prediction settings.
BibTeX
@inproceedings{
daumé2018residual,
title={{RESIDUAL} {LOSS} {PREDICTION}: {REINFORCEMENT} {LEARNING} {WITH} {NO} {INCREMENTAL} {FEEDBACK}},
author={Hal Daumé III and John Langford and Paul Mineiro and Amr Sharaf},
booktitle={International Conference on Learning Representations},
year={2018},
url={https://openreview.net/forum?id=HJNMYceCW},
}