LSTM-MHSA-Enhanced Deep Reinforcement Learning for Accurate Gait Control in Human Musculoskeletal Model

Abstract

Modeling and controlling the musculoskeletal system are crucial for understanding human motor functions, optimizing human-robot interaction, and developing embodied intelligence. However, existing musculoskeletal models are mainly limited to specific body parts and muscle groups, and still face challenges in large-scale muscle coordination and the generation of diverse movements. In this study, we propose a musculoskeletal deep reinforcement learning (DRL) control model. This model integrates a Long Short-Term Memory (LSTM) network and a Multi-Head Self-Attention (MHSA) mechanism into the Proximal Policy Optimization (PPO) algorithm. The LSTM-MHSA-enhanced PPO control approach generates accurate muscle activation, motion trajectories, and torque control strategies to precisely control and replicate diverse human gaits based on target joint movements. Experimental results demonstrate that this LSTM-MHSA-enhanced PPO algorithm significantly improves the model accuracy compared to the traditional PPO algorithm, with a 43.75% and 34.14% reduction in Mean Absolute Error (MAE) for walking and running tasks, respectively. Furthermore, for complex tasks such as striking and dancing, the MAE decreases by 46.97% and 41.78%, respectively. These findings highlight that integrating LSTM and MHSA into PPO algorithm not only enhances gait simulation accuracy but also improves the model’s generalization capability, particularly for complex motion patterns. This research provides an efficient tool for motion simulation and gait analysis, advancing the development of human musculoskeletal control systems.