Promoting Quality and Diversity in Population-based Reinforcement Learning via Hierarchical Trajectory Space Exploration
Jiayu Miao, Tianze Zhou, Kun Shao, Ming Zhou, Weinan Zhang, Jianye Hao, Yong Yu, Jun Wang
Abstract
Quality Diversity (QD) algorithms in population-based reinforcement learning aim to optimize agents' returns and diversity among the population simultaneously. It is conducive to solving exploration problems in reinforcement learning and potentially getting multiple good and diverse strategies. However, previous methods typically define behavioral embedding in action space or outcome space, which neglect trajectory characteristics during the execution process. In this paper, we introduce a trajectory embedding model trained by Variational Autoencoder with similarity constraint to characterize trajectory features. Based on that, we propose a hierarchical trajectory-space exploration (HTSE) framework using Determinantal Point Processes (DPP) to generate high-quality and diverse solutions in the selection and mutation process. The experimental results show that our HTSE method effectively completes several simulated tasks, outperforming other Quality-Diversity Reinforcement Learning algorithms.
BibTeX
@inproceedings{icra2022_promotingquality,
title = {Promoting Quality and Diversity in Population-based Reinforcement Learning via Hierarchical Trajectory Space Exploration},
author = {Jiayu Miao and Tianze Zhou and Kun Shao and Ming Zhou and Weinan Zhang and Jianye Hao and Yong Yu and Jun Wang},
booktitle = {ICRA 2022},
year = {2022}
}