Toward Human Preference Optimization for Vision-Language-Action Models: A Pilot Study on the Limits of Imitation Learning

Abstract

Vision-Language-Action (VLA) models trained via imitation learning have achieved impressive results on robotic manipulation, yet their performance degrades significantly on complex, multi-step tasks. We evaluate NVIDIA GR00T N1.6, a state-of-the-art cross-embodiment VLA model (~1.09B parameters), on the SimplerEnv Fractal benchmark to systematically identify where imitation learning falls short. We conduct closed-loop evaluation across six manipulation tasks of increasing complexity using a Google Robot embodiment, with 200 episodes per task. Our results reveal a stark performance gap: simple tasks such as picking a can achieve 90.0% success rate, while complex sequential tasks such as placing an object in a closed drawer achieve only 4.5%. Average episode time further confirms this — simple tasks complete in under 3 seconds, while complex tasks approach the maximum step timeout, indicating the policy fails to make meaningful progress. We identify three failure modes driving this degradation: absence of recovery behaviors, compounding distribution shift over long horizons, and inability to optimize trajectory quality beyond mimicking demonstrations. Based on these findings, we propose Human Preference Optimization (HPO) as a post-training strategy — leveraging human trajectory rankings and reinforcement learning to refine VLA policies beyond what demonstration data alone can teach.