ARC: Robots Adaptive Risk-aware Robust Control via Distributional Reinforcement Learning

Abstract

Locomotion in robots remains an unsolved challenge, particularly for those with complex structures and dynamic environments. Consequently, the control systems for such robots must place greater emphasis on risk mitigation and safety considerations to ensure reliable and stable operation. Existing studies have explicitly incorporated risk factors into policy training, but lacked the ability to adaptively adjust the risk sensitivity for hazardous environments. This deficiency impacts the agent’s exploration during training and thus fails to select the optimal action. We innovatively introduce Adaptive Risk-aware Control (ARC) policies based on Distributional Reinforcement Learning (Dist.RL), a novel framework that dynamically adjusts risk sensitivity levels in response to changing environmental conditions. Our approach uniquely integrates two key components: (1) the Inter Quartile Range (IQR) for quantifying intrinsic environmental uncertainty, and (2) Random Network Distillation (RND) for evaluating parameter uncertainty. This dual-mechanism architecture represents a significant advancement in risk assessment methodologies. Simulations conducted on a variety of robots have demonstrated that our method achieves significantly more robust performance compared to other approaches. Furthermore, sim2real validation on a humanoid robot confirms the practical viability of our approach.