Hand-Gesture Based Robot Tracking and Teaching Framework: Demonstration in Surface Polishing Task

Abstract

Modern manufacturing is rapidly shifting toward high-mix, low-volume (HMLV) production, which requires flexible and intuitive robot programming methods. However, conventional approaches such as teach pendants or external trackers remain complex and inefficient for non-expert users. This paper presents a hand-gesture based tracking and teaching framework that enables intuitive and no-code robot programming for industrial applications. The system estimates a stable local coordinate frame from robust hand-joint landmarks and employs visual servoing with an eye-in-hand RGB-D camera to continuously and stably track the user's hand. A finger padding filter is introduced to enhance depth estimation at the fingertips, ensuring reliable 3D point extraction for trajectory definition. On top of low-level pose tracking, a gesture recognition model combined with a finite state machine allows non-expert users to design diverse robot programs entirely through hand gestures, without coding. The proposed framework was first validated through controlled experiments, demonstrating improved teaching accuracy and reduced errors compared to a fixed-camera baseline. Its practical utility was then evaluated in a representative application scenario, where the complete workflow of a surface finishing task such as polishing was carried out solely via hand gestures. A usability study further confirmed its effectiveness, showing high user acceptance and reduced workload. Overall, the framework provides an accessible and generalizable interface for robot programming, supporting flexible deployment in high-mix, low-volume manufacturing.