Offline Reinforcement Learning for Quadrotor Control: Overcoming the Ground Effect

Abstract

Applying Reinforcement Learning to solve real-world optimization problems presents significant challenges because of the large amount of data normally required. A popular solution is to train the algorithms in a simulation and transfer the weights to the real system. However, sim-to-real approaches are prone to fail when the Reality Gap is too big, e.g. in robotic systems with complex and non-linear dynamics. In this work, we propose the use of Offline Reinforcement Learning as a viable alternative to sim-to-real policy transfer to address such instances. On the example of a small quadrotor, we show that the ground effect causes problems in an otherwise functioning zero-shot sim-to-real framework. Our sim-to-real experiments show that, even with the explicit modelling of the ground effect and the employing of popular transfer techniques, the trained policies fail to capture the physical nuances necessary to perform a real-world take-off maneuver. Contrariwise, we show that state-of-the-art Offline Reinforcement Learning algorithms represent a feasible, reliable and sample efficient alternative in this use case.