Adaptive Neural Uncalibrated Visual Servo with Zero-shot Transfer of Extrinsics and Scenes

Abstract

Deploying visual servo controller to novel scenes with uncertain parameters requires additional manual effort for calibration. Traditional methods tackle this problem by online estimating the Jacobian matrix. However, they struggle in challenging scenes due to intrinsic limitations. For instance, image-based uncalibrated visual servo requires tracking a fixed set of points, which is impractical in texture-less scenes. Position-based uncalibrated visual servo necessitates absolute scale of translation, which requires depth sensor or model-based pose estimator, introducing extra hardware cost or model complexity. Recent advances in neural network-based visual servoing have shown improvement in convergence, precision and generalization compared to traditional methods. However, the uncalibrated neural visual servo remains underexplored. In this paper, we propose a structured Jacobian estimator for neural-based visual servo controller, enabling zero-shot transfer to novel environments with unknown extrinsic and scene scale. Stability of pose error is analyzed under the bounded calibration error assumption. Moreover, we propose an automatic control gain scheduler to accelerate the convergence while maintaining high success rate and precision. The scheduling behavior is analyzed through greedy optimal control. Our method is validated with simulated and real-world experiments.