Meta-Reinforcement Learning With Evolving Gradient Regularization

Abstract

Deep reinforcement learning (DRL) typically requires reinitializing training for new tasks, limiting its generalization due to isolated knowledge transfer. Meta-reinforcement learning (Meta-RL) addresses this by enabling rapid adaptation through prior task experiences, yet existing gradient-based methods like MAML suffer from poor out-of-distribution performance due to overfitting narrow task distributions. To overcome this limitation, we propose Evolving Gradient Regularization MAML (ER-MAML). By integrating evolving gradient regularization into the MAML framework, ER-MAML optimizes meta-gradients while constraining adaptation directions via a regularization policy. This dual mechanism prevents overparameterization and enhances robustness across diverse task distributions. Experiments demonstrate ER-MAML outperforms state-of-the-art baselines by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$14.6\%$</tex-math></inline-formula> in out-of-distribution success rates. It also achieves strong online adaptation performance in the MetaWorld benchmark. These results validate ER-MAML's effectiveness in improving meta-RL generalization under distribution shifts.