ZeroBot: Learning From Scratch in Minutes With Generative Real2Sim
Ivan Kapelyukh, Xiaohan Zhang, Stephen James, Laura Herlant, Edward Johns
Abstract
We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning. Project page: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">zerobot-rl.github.io</uri>
BibTeX
@inproceedings{ral2026_zerobotlearningf,
title = {ZeroBot: Learning From Scratch in Minutes With Generative Real2Sim},
author = {Ivan Kapelyukh and Xiaohan Zhang and Stephen James and Laura Herlant and Edward Johns},
booktitle = {RA-L 2026},
year = {2026}
}