Point Cloud Based Visual Planning and Servoing for Autonomous Vine Pruning

Abstract

This paper addresses vision based autonomous vine pruning with robots. The complex structure of vines makes visual servoing difficult due to challenges in feature extraction and 3D pose estimation. A novel approach interweaving visual servoing and vision based planning is proposed. An online planner features a Nonlinear Model Predictive Control to compute on-the-fly suitable 3D waypoints. These are navigated by a Position Based Visual Servoing integrating Iterative Closest Point based point-cloud alignment. Qualitative and quantitative evaluations are conducted on live experiments where a Franka Emika manipulator embedding a eye-in-hand stereo camera performs a sequence of pruning actions on seven increasingly complex vine stocks, in spite of unexpected motion of the vine and/or the robot's base. Outdoor experiments are also carried out in strong wind conditions.