AffPose: An Integrated RGB-Based Framework for Simultaneous Pose Estimation and Affordance Detection in Robotic Tool Manipulation

Abstract

Enabling robots to perform tool manipulation like humans remains a great challenge. A semantic understanding of tool affordances and precise spatial localization is essential for this task. Conventional methods relying on RGB-D cameras for affordance detection and tool manipulation have been proven inadequate for low-texture, reflective, or light-absorbing black tools. We present AffPose, an integrated RGB-based framework that synergistically combines affordance detection and pose estimation. First, our Directional-Enhanced Mask R-CNN significantly improves edge orientation perception for affordance segmentation. Second, a novel masked attention mechanism leverages predicted affordance regions to guide the pose estimation network, reducing redundant feature processing. Third, we establish the first comprehensive Affordance-Pose dataset with synchronized ground-truth annotations for the affordance mask and 6D pose. Extensive experiments demonstrate that our framework achieves outperformance in affordance segmentation and pose estimation tasks. Real-world experiments also showcase the reliability of our method in accomplishing human-like manipulation tasks with daily tools.