Heterogeneous Embodied Multi-Agent Collaboration

Abstract

Multi-agent embodied tasks have been studied in indoor visual environments, but most of the existing research focuses on homogeneous multi-agent tasks. Heterogeneous multi-agent tasks are common in real-world scenarios, and the collaboration strategy among heterogeneous agents with different capabilities is a challenging and important problem to be solved. To study collaboration among heterogeneous agents, we propose the heterogeneous multi-agent tidying-up task, in which heterogeneous agents collaborate with others to detect misplaced objects and place them in reasonable locations. This is a demanding task since it requires agents to make the best use of their different capabilities to conduct reasonable task planning and allocation. We build a benchmark dataset based on ProcTHOR-10K. We propose the hierarchical decision model based on misplaced object detection, reasonable receptacle prediction and handshake-based group communication mechanism. Extensive experiments are conducted to demonstrate the effectiveness of the proposed model. The experimental videos can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://hetercol.github.io/</uri> .