Variance-Aware Off-Policy Evaluation with Linear Function Approximation
Yifei Min, Tianhao Wang, Dongruo Zhou, Quanquan Gu
Abstract
We study the off-policy evaluation (OPE) problem in reinforcement learning with linear function approximation, which aims to estimate the value function of a target policy based on the offline data collected by a behavior policy. We propose to incorporate the variance information of the value function to improve the sample efficiency of OPE. More specifically, for time-inhomogeneous episodic linear Markov decision processes (MDPs), we propose an algorithm, \texttt{VA-OPE}, which uses the estimated variance of the value function to reweight the Bellman residual in Fitted Q-Iteration. We show that our algorithm achieves a tighter error bound than the best-known result. We also provide a fine-grained characterization of the distribution shift between the behavior policy and the target policy. Extensive numerical experiments corroborate our theory.
BibTeX
@inproceedings{
min2021varianceaware,
title={Variance-Aware Off-Policy Evaluation with Linear Function Approximation},
author={Yifei Min and Tianhao Wang and Dongruo Zhou and Quanquan Gu},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=FKmcLhJ4mn}
}