CDA-MBPO: Corrected Data Aggregation for Model-Based Policy Optimization

Abstract

Model-based reinforcement learning has shown promise in sample efficiency but suffers from errors accumulated during multi-step model sampling. To tackle this issue, we propose corrected data aggregation for model-based policy optimization. This approach involves aligning simulated trajectories with their real counterparts from random starting states and with varying sampling lengths to create paired real-simulated samples. The R-Q discriminator is incorporated to assess the quality of the simulated samples by computing the R-Q difference, modeled as a Gaussian distribution within each paired sample. We update the Q network and the dynamics model using all real samples and the simulated samples whose R-Q difference fall below a predefined threshold. The experimental results demonstrate that our method outperforms state-of-the-art model-based methods in sample efficiency and asymptotic performance across challenging tasks. Our code is available at https://github.com/duxin0618/CDA-MBPO.