Continual Auxiliary Task Learning
Matthew K McLeod, Chunlok Lo, Matthew Kyle Schlegel, Andrew Jacobsen, Raksha Kumaraswamy, Martha White, Adam M White
Abstract
Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems. A variety of off-policy learning algorithms have been developed to learn such predictions, but as yet there is little work on how to adapt the behavior to gather useful data for those off-policy predictions. In this work, we investigate a reinforcement learning system designed to learn a collection of auxiliary tasks, with a behavior policy learning to take actions to improve those auxiliary predictions. We highlight the inherent non-stationarity in this continual auxiliary task learning problem, for both prediction learners and the behavior learner. We develop an algorithm based on successor features that facilitates tracking under non-stationary rewards, and prove the separation into learning successor features and rewards provides convergence rate improvements. We conduct an in-depth study into the resulting multi-prediction learning system.
BibTeX
@inproceedings{
mcleod2021continual,
title={Continual Auxiliary Task Learning},
author={Matthew K McLeod and Chunlok Lo and Matthew Kyle Schlegel and Andrew Jacobsen and Raksha Kumaraswamy and Martha White and Adam M White},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=EpL9IFAMa3}
}