Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds

Zhiyong Wang; Dongruo Zhou; John C.S. Lui; Wen Sun

ICLR 2025poster5 citations

Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds

Zhiyong Wang, Dongruo Zhou, John C.S. Lui, Wen Sun

Abstract

Learning a transition model via Maximum Likelihood Estimation (MLE) followed by planning inside the learned model is perhaps the most standard and simplest Model-based Reinforcement Learning (RL) framework. In this work, we show that such a simple Model-based RL scheme, when equipped with optimistic and pessimistic planning procedures, achieves strong regret and sample complexity bounds in online and offline RL settings. Particularly, we demonstrate that under the conditions where the trajectory-wise reward is normalized between zero and one and the transition is time-homogenous, it achieves nearly horizon-free and second-order bounds.

reinforcement learning theorymodel-based reinforcement learning

BibTeX

@inproceedings{
wang2025modelbased,
title={Model-based {RL} as a Minimalist Approach to Horizon-Free and Second-Order Bounds},
author={Zhiyong Wang and Dongruo Zhou and John C.S. Lui and Wen Sun},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=txD9llAYn9}
}