ShapeAfford: Reconstructing 3D Shape With Manipulation Affordance via Geometry-Affordance Synergy

Abstract

To facilitate robot manipulation tasks, we propose ShapeAfford that reconstructs 3D object models with per-point affordance annotations from multi-view images and textual instructions. By integrating geometric modeling with affordance reasoning into an end-to-end framework, ShapeAfford leverages the mutual constraints and the interplay between object geometry and its affordance to overcome inherent limitations in affordance prediction such as point cloud sparsity and viewpoint sensitivity. We also construct a novel dataset containing 3D models represented as point clouds and meshes, affordance annotations, and natural language instructions, enabling the model to reconstruct 3D objects along with corresponding affordances. Experimental results demonstrate that ShapeAfford outperforms existing approaches on both 3D reconstruction and affordance prediction tasks. Furthermore, we show that our method can be directly used in downstream robot manipulation tasks in real-world environments.