VRobotix: A Scalable and Cost-Effective Virtual-Reality-Based Robotic Manipulation Dataset Generation Framework

Abstract

Large-scale, diverse datasets are essential for training robust learning-based robotic manipulation models; however, their acquisition typically requires controlled environments and specialized hardware in research laboratories. This paper presents VRobotix, a virtual reality (VR)-based framework that enables cost-effective and scalable robotic dataset generation through immersive human-in-the-loop control within a physics-accurate robot simulation. By leveraging off-the-shelf VR headsets (e.g., Oculus Quest 3), VRobotix eliminates the need for physical robots while supporting a URDF-compatible, physics-based simulator that accommodates adaptable robotic platforms and egocentric control interfaces, including handheld controllers and body posture tracking. Benefiting from the physics-based simulation, a unique contribution of VRobotix is the replay module, which can regenerate synchronized multi-modal dataset (kinematic states, RGB-D streams) with multiple dataset formats based on the replayable trajectory, supporting various robotic applications. Additionally, an imitation learning module is developed to train control policies using the data collected by VRobotix. Experiments on three initial tasks—pushing, grasping, and stacking—demonstrate a high data collection success rate, averaging 92.0%. Furthermore, policies trained on just 50 trials achieve a 100% task success rate. VRobotix reduces infrastructure costs while generating ROS-compatible datasets, democratizing scalable robotic data acquisition.