SGFAM: Semantic and Geometric Features Aggregation for Dense Shape Matching in Generalizable Robotic Manipulation

Abstract

To automate processes like polishing or cleaning at scale, robots must be able to adapt learned skills to new object instances without manual reprogramming. Applications requiring tool-surface interactions face a significant challenge in transferring manipulation strategies to novel objects due to substantial shape and appearance variations. Robust, generalized dense shape correspondence is essential for solving this problem. We present SGFAM, a zero-shot pipeline integrating pre-trained vision foundation models and geometric encoders via functional maps. Unlike prior works that rely on simple averaging, we introduce 1) an alignment-based feature aggregation to prioritize optimal viewing angles, and 2) Kernel PCA fusion to preserve non-linear descriptor manifolds. Our evaluations demonstrate that this approach not only outperforms state-of-the-art baselines but also enables lightweight vision backbones to achieve matching precision comparable to larger models. We validate SGFAM experimentally by successfully transferring continuous surface paths in real-world industrial and household robotic scenarios without requiring any finetuning.