Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stopping
David Snyder, Asher James Hancock, Apurva Badithela, Emma Dixon, Patrick Miller, Rares Andrei Ambrus, Anirudha Majumdar, Masha Itkina
Abstract
Imitation learning has enabled robots to perform complex, long-horizon tasks in challenging dexterous manipulation settings. As new methods are developed, they must be rigorously evaluated and compared against corresponding baselines through repeated evaluation trials. However, policy comparison is fundamentally constrained by a small feasible sample size (e.g., 10 or 50) due to significant human effort and limited inference throughput of policies. This paper proposes a novel statistical framework for rigorously comparing two policies in the small sample size regime. Prior work in statistical policy comparison relies on batch testing, which requires a fixed, pre-determined number of trials and lacks flexibility in adapting the sample size to the observed evaluation data. Furthermore, extending the test with additional trials risks inducing inadvertent p-hacking, undermining statistical assurances. In contrast, our proposed statistical test is
BibTeX
@inproceedings{rss2025_isyourimitationl,
title = {Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stopping},
author = {David Snyder and Asher James Hancock and Apurva Badithela and Emma Dixon and Patrick Miller and Rares Andrei Ambrus and Anirudha Majumdar and Masha Itkina and Haruki Nishimura},
booktitle = {RSS 2025},
year = {2025}
}