Topology Decoupled All-reduce Algorithm

Abstract

With the advancement of deep learning, network communication has become the most critical factor in model training. Especially, the all-reduce operation can comprise over 70% of the cumulative training duration as a pivotal component within data parallelism. However, existing all-reduce algorithms often perform poorly in complex network topologies, and there is currently no universal and straightforward all-reduce algorithm that can effectively adapt to diverse topological structures.In this work, we propose a topology decoupled all-reduce algorithm. We decouple the network into multiple tree substructures, select the trees with the smallest heights, and then split the data to perform aggregate communication within these selected structures. This approach significantly reduces the number of communications and enhances efficiency. Experimental results show that our topology decoupled all-reduce algorithm reduces communication time compared to NCCL’s by 42.9% and enhances end-to-end training efficiency by 11.7%.