PLOP: Particle Filtering for Learning Object Physics from Robot Interaction Videos

Abstract

Learning the dynamics of deformable objects, such as dough or a sponge, from RGB-D videos is challenging due to insufficient visual cues and complex deformations. We introduce PLOP (Particle Filtering for Learning Object Physics), a novel framework to learn the dynamics model of deformable objects using a particle filter over 3D Gaussians. Our method learns (1) a dynamics function to predict the object state at the next time step and (2) a resampling function to split and merge Gaussians to handle complex deformations such as cutting. Within PLOP, we propose I2N (Implicit Particle Interaction Network), a dynamics model that leverages a mixed particle-grid representation inspired by the Material Point Method (MPM). By transferring particle features to grid nodes, solving for grid dynamics, and then projecting solutions back to particles, our approach avoids explicit pairwise interaction reasoning between particles and significantly reduces computational cost when the number of particles is large. While PLOP is applicable to general robot-object interactions, we evaluate it on cutting sequences in both simulation and the real world, which induce challenging topological changes and expose previously occluded surfaces. On these benchmarks, PLOP achieves a 53.15% improvement in 3D reconstruction accuracy and a 6.84% improvement in 2D reconstruction accuracy on the simulation benchmark, as well as 28.41% and 24.45% improvements in 3D and 2D reconstruction metrics, respectively, on the real-world dataset.