Fault Tolerant Multi-Agent Learning with Adversarial Budget Constraints

Abstract

We study robustness to agent malfunctions in cooperative multi-agent reinforcement learning (MARL), a failure mode that is critical in practice yet underexplored in existing theory. We introduce MARTA, a plug-and-play robustness layer that augments standard MARL algorithms with a {\fontfamily{cmss}\selectfont Switcher}–{\fontfamily{cmss}\selectfont Adversary} mechanism which selectively induces malfunctions in performance-critical states. This formulation defines a fault-switching $(N+2)$-player Markov game in which the {\fontfamily{cmss}\selectfont Switcher} chooses when and which agent fails, and the {\fontfamily{cmss}\selectfont Adversary} controls the resulting faulty behaviour via random or worst-case policies. We develop a Q-learning-type scheme and show that the associated Bellman operator is a contraction, yielding existence and uniqueness of the minimax value, convergence to a Markov perfect equilibrium. MARTA integrates seamlessly with MARL algorithms without architectural modification and consistently improves robustness across Traffic Junction (TJ), Level-Based Foraging (LBF), MPE SimpleTag, and SMAC (v2). In these domains, MARTA achieves large gains in final performance of up to \textbf{116.7\%} in SMAC, \textbf{21.4\%} in MPE SimpleTag, and \textbf{44.6\%} in LBF, while significantly reducing failure rates under train–test mismatched fault regimes. These results establish MARTA as a theoretically grounded and practically deployable mechanism for fault-tolerant MARL.