Preslav Aleksandrov

DES-LOC: Desynced Low Communication Adaptive Optimizers for Foundation Models

Scaling foundation model training with Distributed Data Parallel~(DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize model parameters only and cannot be trivially applied to adaptive optimizers due to additional optimizer states.…