Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning

Na Li; Zewu Zheng; Wei Ni; Hangguan Shan; Wenjie Zhang; Xinyu Li

NeurIPS 2025poster0 citations

Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning

Na Li, Zewu Zheng, Wei Ni, Hangguan Shan, Wenjie Zhang, Xinyu Li

Abstract

Multi-agent reinforcement learning (MARL), as a thriving field, explores how multiple agents independently make decisions in a shared dynamic environment. Due to environmental uncertainties, policies in MARL must remain robust to tackle the sim-to-real gap. We focus on robust two-player zero-sum Markov games (TZMGs) in offline settings, specifically on tabular robust TZMGs (RTZMGs). We propose a model-based algorithm (*RTZ-VI-LCB*) for offline RTZMGs, which is optimistic robust value iteration combined with a data-driven Bernstein-style penalty term for robust value estimation. By accounting for distribution shifts in the historical dataset, the proposed algorithm establishes near-optimal sample complexity guarantees under partial coverage and environmental uncertainty. An information-theoretic lower bound is developed to confirm the tightness of our algorithm's sample complexity, which is optimal regarding both state and action spaces. To the best of our knowledge, RTZ-VI-LCB is the first to attain this optimality, sets a new benchmark for offline RTZMGs, and is validated experimentally.

robust Markov gamesself-playdistribution shiftmodel uncertaintyreinforcement learning

BibTeX

@inproceedings{
li2025sampleefficient,
title={Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning},
author={Na Li and Zewu Zheng and Wei Ni and Hangguan Shan and Wenjie Zhang and Xinyu Li},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=xVsC90U8yl}
}