AERAS: Adaptive Experience Replay with Attention-Based Sequence Embedding for Improved Multi-Agent Reinforcement Learning

Abstract

Multi-agent systems in non-stationary environments face challenges due to rapidly changing dynamics, leading to quick obsolescence of experiences in the replay buffer. To address this, we propose the Adaptive Experience Replay with Attention-Based Sequence Embedding (AERAS) framework, which integrates sequence embedding with an attention mechanism to prioritize experiences based on their relevance. By assigning adaptive weights, AERAS emphasizes relevant experiences while diminishing the impact of outdated ones, enhancing efficiency and learning performance in multi-agent reinforcement learning. Evaluations on the StarCraft II Multi-Agent Challenge and Google Research Football environments show that AERAS consistently outperforms state-of-the-art methods, achieving faster convergence and higher win rates. Ablation studies confirm the essential roles of sequence embedding and attention mechanisms in boosting AERAS's robustness and adaptability, underscoring its effectiveness in managing non-stationary environments within multi-agent systems.