Integrating Artificial Vision and Wearable Robotics: Adaptive Assistance Enabled by Manipulation Context Awareness

Abstract

Occupational exoskeletons are emerging as a promising solution for industrial applications, providing support to reduce fatigue and the risk of musculoskeletal disorders. One of the main challenges limiting their widespread adoption is that most existing devices cannot deliver real-time, adaptable, and context-aware assistance. This paper presents the first fully vision-driven control strategy for a bimanual upper-limb soft exoskeleton, enabling adaptive assistance during industrial tool manipulation. The approach integrates three modules: tool recognition and segmentation, hand tracking with gesture recognition, and a fusion layer that ensures reliable understanding of the manipulation context. This allows modulation of lifting assistance in real time according to the weight of the grasped object. Experiments with human participants demonstrated that the proposed approach reduces biceps activation by more than 50% compared to the no-support condition, while operating in real time on embedded hardware. The method is robust to hand–object occlusions, camera repositioning, and dynamic environments, demonstrating its practicality for industrial deployment. Overall, this work establishes vision-based control as a scalable solution for ergonomic, adaptive exoskeletons that enhance safety and productivity in demanding workplaces.