Two-stage Learning Framework Combining Joint-level Reinforcement Learning and Muscle-level Adaptation for Musculoskeletal Locomotion

Abstract

Animal musculoskeletal systems are renowned for their ability to dynamically regulate stiffness and achieve energy-efficient motion. Being inspired by the biological control structure, this study presents a hybrid control framework that utilizes two-stage learning processes for body movement planning and muscle force computation. This methodology simplifies the learning process under joint redundancy and muscle redundancy. Then it enhances the interpretability of the resultant generated behaviors. The framework incorporates a reinforcement learning (RL)-trained joint controller to optimize joint torques, in conjunction with an LSTM-based muscle controller that translates these torques into muscle activations. Two control variants are proposed: One is prioritizing energy efficiency and the other is enhancing adaptability to environmental perturbations through co-contraction control. Validation with MuJoCo physics simulations demonstrates the framework’s capacity to autonomously learn and refine different gait modes without dependence on external motion datasets. The second variant demonstrates superior robustness and energy efficiency compared to conventional motor-driven models. This framework contributes to the enhancement of adaptability in complex scenarios dealing with the redundancy problem of musculoskeletal system coordination and holds potential for the development of bio-inspired locomotion control through the optimization of muscle activity composition.