MEPE: A Minimalist Ensemble Policy Evaluation Operator for Deep Reinforcement Learning

Abstract

Ensemble deep reinforcement learning (DRL) is a popular approach to mitigate the risk of overspecialization to a particular distribution and improve performance. However, it is commonly plagued by high computational resource requirements that arise from the introduction of multiple value and policy functions. To avoid the notorious resource consumption issue, we design a simple yet effective ensemble policy evaluation operator, termed Minimalist Ensemble Policy Evaluation (MEPE), which leverages a modified dropout operator combined with the Bellman equation. MEPE is equivalent to integrating multiple models into a single model. The MEPE operator holds ensemble property by keeping the dropout consistency of both sides of the Bellman equation and can be combined with any DRL algorithms if they have a policy evaluation phase. To verify the MEPE’s ability, we perform experiments on both low-dimensional and high-dimensional environments, which presents that the algorithms combined with the MEPE operator outperform or achieve a similar level of performance as the current state-of-the-art ensemble methods and model-free methods without increasing additional computational resource costs. To the best of our knowledge, our work is the first that applies the dropout operator both in low-dimensional control tasks and high-dimensional games. Our code is available at https://github.com/sweetice/MEPE.