Heterogeneous Multi-Agent Learning in Isaac Lab: Scalable Simulation for Robotic Collaboration

Abstract

Multi-Agent Reinforcement Learning (MARL) plays a crucial role in robotic coordination and control, yet existing simulation environments often lack the fidelity and scalability needed for real-world applications. In this work, we extend Isaac Lab to support efficient training of both homogeneous and heterogeneous multi-agent robotic policies in high-fidelity physics simulations. Our contributions include the development of diverse MARL environments tailored for robotic coordination, integration of Heterogeneous Agent Reinforcement Learning (HARL) algorithms, and a scalable GPU-accelerated framework optimized for large-scale training. We evaluate our framework using two state-of-the-art MARL algorithms—Multi-Agent Reinforcement Learning with Proximal Policy Optimization (MAPPO) and Heterogeneous Agent Reinforcement Learning with Proximal Policy Optimization (HAPPO)—across several robotic tasks. Our results confirm the feasibility of training heterogeneous agents in high-fidelity environments while maintaining the scalability and performance benefits of Isaac Lab. By advancing realistic multi-agent learning at scale, our work lays a foundation for more MARL research in physics-driven robotics. The source code and demonstration videos are available at https://some45bucks.github.io/IsaacLab-HARL/.