Hierarchical Tri-Manual Planning for Vision-Assisted Fruit Harvesting with Quadrupedal Robots

Abstract

This paper addresses the challenge of developing a multi-arm quadrupedal robot capable of efficiently harvesting fruit in complex, natural environments. To overcome the inherent limitations of traditional bimanual manipulation, we introduce the first three-arm quadrupedal robot LocoHarv3, that builds on top of the Spot quadruped, and propose a novel hierarchical tri-manual planning approach for automated fruit harvesting with collision-free trajectories between the built-in end-effector of Spot and our custom-made bimanual manipulator. Our comprehensive semi-autonomous framework integrates teleoperation, supported by LiDAR-based odometry and mapping, with learning-based visual perception for accurate fruit detection and pose estimation. Validation is conducted through a series of controlled indoor experiments using motion capture and extensive field tests in natural settings. Results demonstrate a 90 % success rate in in-lab settings with a single attempt, and field trials further verify the system's robustness and efficiency in more challenging real-world environments.