AutoTrialGen: Automated Data Generation From Few Human Demonstrations via Trajectory Annotation and Simulation Trials

Abstract

While imitation learning is a powerful paradigm for teaching robots complex manipulation skills, its effectiveness is often bottlenecked by the need for large-scale, human-collected datasets. This paper presents AutoTrialGen, an automated framework designed to generate large and diverse datasets of successful simulated demonstrations through trial-and-error, from only a few human demonstrations. Initially, AutoTrialGen employs a vision-language model to automatically decompose unprocessed human demonstrations into a library of object-centric and reusable skill primitives. Within a simulated environment, it then intelligently composes these primitives to solve new task instances, employing a novel weighted manipulability selection mechanism to ensure the quality and efficiency of the generated trajectories. Policies trained using the proposed sim-augmented data demonstrate improved performance and enhanced data efficiency across six diverse and challenging real-world manipulation tasks, compared with the baseline method.