DASP: Hierarchical Offline Reinforcement Learning via Diffusion Autodecoder and Skill Primitive

Abstract

Offline reinforcement learning strives to enable agents to effectively utilize pre-collected offline datasets for learning. Such an offline setup tremendously mitigates the problems of online reinforcement learning algorithms in real-world applications, particularly in scenarios where interactions are constrained or exploration is costly. The learned strategy, on the other hand, has a distributional bias with respect to the behavioral strategy, which consequently leads to the problem of extrapolation error for out-of-distribution actions. To mitigate this problem, in this paper, we adopt a hierarchical offline reinforcement learning framework that extracts recurrent and spatio-temporally extended primitive skills from offline data before using them for downstream task learning. Besides, we introduce an autodecoder conditional diffusion model to characterize low-level strategy decoding. Such a deep learning generative model enables the reduction of action primitives for the strategy space, which is then used to learn high-level task strategy-guided primitives via the offline learning algorithm IQL. Experimental results and ablation studies on D4RL benchmark tasks (Antmaze, Adroit and Kitchen) demonstrate that our approach achieves SOTA performance in most tasks.