Exploring Haptic Augmentation and Language Design for Smartphone-Based Teleoperation

Abstract

Smartphone-based teleoperation is gaining traction as a versatile remote control solution, using widely available hardware to provide a portable and scalable interface for telerobotics. However, a crucial limitation of such an approach is the lack of effective haptic feedback, which restricts accuracy and increases operator workload. While smartphones offer a low-entry barrier as well as both portability and scalability, current interfaces rely almost exclusively on visual cues. To address this gap, we investigate the use of symbolic haptic feedback delivered through an unmodified mobile device to support remote manipulation tasks. We designed a combined teleoperation task that integrates object sorting and peg-in-hole insertion, embedding five candidate haptic cues (i.e., contact, gripper state, alignment, error boundary, and motion initiation). A within-subjects study with 16 participants compared visual-only and visual-plus-haptic conditions. Results show that haptic augmentation reduced total errors by 42% and significantly lowered perceived workload. Continuous cues for alignment and error boundaries achieved the highest recognition rates of 94% and 81%, respectively, while brief state cues were less reliably interpreted. Post-task interviews highlighted user preference for simple, continuous, and intense signals in visually ambiguous scenarios. Our findings provide new design guidelines for haptic cue prioritisation and encoding strategies.