Reinforcement Learning-Based Autonomous Control Methodology of Hydraulic Excavators

Abstract

The automation of hydraulic excavators is significant for enhancing productivity and safety in uncertain and dynamic environments. Achieving autonomous operation requires advanced control strategies capable of handling system constraints, nonlinear hydraulic dynamics, and complex environmental interactions. This study proposes a reinforcement learning (RL)-based methodology to perform a complete excavation cycle by controlling proportional valves. A comprehensive joint simulation tool is developed, in which a hydraulic system model is detailed based on a real machine, and it is integrated with an excavator mechanism and working environment to create a realistic interaction environment for RL training. The RL agent, trained using Proximal Policy Optimization (PPO), incorporates a customized reward shaping method that ensures operational safety and accuracy, considering constraints such as pump flow saturation and geometric constraints. In addition, an Adaptive Control Frequency (ACF) method is developed to enhance training efficiency by dynamically adjusting the control frequency based on task complexity. Comparative validations demonstrate the RL agent’s ability to successfully complete a full excavation cycle, satisfy operational constraints, and generalize across varying initial conditions and valve responses. Furthermore, the controller operates effectively in a soil environment despite being trained without soil, demonstrating robustness to uncertain, time-varying loads.