On the Performance of Temporal Difference Learning With Neural Networks

HAOXING TIAN; Ioannis Paschalidis; Alex Olshevsky

ICLR 2023poster9 citations

On the Performance of Temporal Difference Learning With Neural Networks

HAOXING TIAN, Ioannis Paschalidis, Alex Olshevsky

Abstract

Neural Temporal Difference (TD) Learning is an approximate temporal difference method for policy evaluation that uses a neural network for function approximation. Analysis of Neural TD Learning has proven to be challenging. In this paper we provide a convergence analysis of Neural TD Learning with a projection onto $B(\theta_0, \omega)$, a ball of fixed radius $\omega$ around the initial point $\theta_0$. We show an approximation bound of $O(\epsilon + 1/\sqrt{m})$ where $\epsilon$ is the approximation quality of the best neural network in $B(\theta_0, \omega)$ and $m$ is the width of all hidden layers in the network.

BibTeX

@inproceedings{
tian2023on,
title={On the Performance of Temporal Difference Learning With Neural Networks},
author={HAOXING TIAN and Ioannis Paschalidis and Alex Olshevsky},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=6JMXLWX68Kj}
}