MoReL: A Generalizable Framework for Dexterous Hand Retargeting via Modular Residual Reinforcement Learning

Abstract

Effective motion retargeting is essential for robotic hands to perform fine-grained teleoperated manipulation. However, existing methods face several key challenges: optimization-based approaches offer accurate reproduction but suffer from high computational latency; learning-based methods provide faster inference but require large-scale datasets; and achieving fine-grained retargeting often compromises hardware adaptability due to task- or hand-specific designs. To this end, we propose Modular Residual Reinforcement Learning (MoReL), a generalizable reinforcement learning framework for dexterous hand retargeting. MoReL decomposes policy learning into finger-specific subpolicies and a residual coordination module, effectively balancing detailed local control with coherent global motion. This architecture enables efficient training from minimal demonstrations without reliance on pretrained networks, while achieving low-latency inference and supporting flexible input modalities. A structured reward formulation further preserves human manipulation nuances and promotes generalization across diverse robotic hands and task scenarios. Extensive experiments validate the effectiveness of our architecture and reward design, demonstrating MoReL's superior performance and cross-platform adaptability in fine-grained dexterous manipulation tasks.