Training Recurrent Neural Networks Online by Learning Explicit State Variables
Somjit Nath, Vincent Liu, Alan Chan, Xin Li, Adam White, Martha White
Abstract
Recurrent neural networks (RNNs) allow an agent to construct a state-representation from a stream of experience, which is essential in partially observable problems. However, there are two primary issues one must overcome when training an RNN: the sensitivity of the learning algorithm's performance to truncation length and and long training times. There are variety of strategies to improve training in RNNs, the mostly notably Backprop Through Time (BPTT) and by Real-Time Recurrent Learning. These strategies, however, are typically computationally expensive and focus computation on computing gradients back in time. In this work, we reformulate the RNN training objective to explicitly learn state vectors; this breaks the dependence across time and so avoids the need to estimate gradients far back in time. We show that for a fixed buffer of data, our algorithm---called Fixed Point Propagation (FPP)---is sound: it converges to a stationary point of the new objective. We investigate the empirical performance of our online FPP algorithm, particularly in terms of computation compared to truncated BPTT with varying truncation levels.
BibTeX
@inproceedings{
Nath2020Training,
title={Training Recurrent Neural Networks Online by Learning Explicit State Variables},
author={Somjit Nath and Vincent Liu and Alan Chan and Xin Li and Adam White and Martha White},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=SJgmR0NKPr}
}