Bimanual Long-Horizon Manipulation Via Temporal-Context Transformer RL

Abstract

Dual-arm robots can perform bimanual long-horizon (LH) manipulation, surpassing the capabilities of single-arm robots. However, bimanual LH tasks are challenging for robot intelligence due to the complexity of long sequence variables and multi-agent interactions. While Multi-Agent Reinforcement Learning (MARL) has shown promising results in agent interactions, these models struggle with sequential LH tasks due to limitations in credit assignment, vanishing memory, and the exploration-exploitation trade-off. This paper introduces a novel dual-arm robot intelligence framework, Temporal-Context Transformer Reinforcement Learning (TC-TRL), which integrates both a hybrid offline-online policy and imitation learning. TC-TRL leverages the attention mechanism to identify relevant temporal-context information from the LH observations space, updating the encoder value function and generating an optimal actions sequence using a decoder module, which uses demonstration guidance during online training. TC-TRL is tested on six bimanual tasks, and its performance is compared against five baseline RLs: MAPPO, HAPPO, IPPO, MAT, and DA-MAT. The results show that TC-TRL outperforms the three PPO-based RLs with an average success rate of 63.46%, 42.23% against MAT, and 30.91% for DA-MAT.