2025
Topology Decoupled All-reduce Algorithm
ICASSP 2025accepted
With the advancement of deep learning, network communication has become the most critical factor in model training. Especially, the all-reduce operation can comprise over 70% of the cumulative training duration as a pivotal component within data parallelism. However, existing all-reduce algorithms o…