Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
Shilong Deng, Zetao Zheng, Hongcai He, Paul Weng, Jie Shao
Abstract
A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly outperform state-of-the-art methods.
BibTeX
@article{Deng_Zheng_He_Weng_Shao_2025, title={Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/33784}, DOI={10.1609/aaai.v39i15.33784}, abstractNote={A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly outperform state-of-the-art methods.}, number={15}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Deng, Shilong and Zheng, Zetao and He, Hongcai and Weng, Paul and Shao, Jie}, year={2025}, month={Apr.}, pages={16244-16252} }