Gotta Scoop 'Em All: Sim-And-Real Co-Training of Graph-Based Neural Dynamics for Long-Horizon Scooping

Abstract

Robotic manipulation of granular objects is crucial in various fields, yet modeling their complex dynamics and diverse physical properties remains challenging. Simulation plays an important role in learning robotic manipulation policies, but it exhibits challenge to accurately model the complex dynamics and physical properties given only visual observations. The difficulty is further compounded in tasks involving intricate contact mechanisms, particularly when using tools with complex shapes like spoons to interact with granular objects, resulting in a significant sim-to-real gap. To address this, we introduce a novel task of scooping all objects out of a storage container using a spoon, which requires sophisticated modeling of multi-object interactions. We propose a unified framework that combines rich simulation data with a small amount of real-world data. Rather than optimizing physical parameters in simulation, we learn a graph-based neural dynamics model in simulation and fine-tune it on real-world data. We then employ a Monte-Carlo Tree Search (MCTS)- based planner to accomplish long-horizon decision-making. Our system successfully scoops out three types of objects, demonstrating its potential for real-world applications. This work highlights the benefits of leveraging both simulation and real-world data to tackle the sim-to-real gap in contact-rich manipulation tasks.