ICML 2022spotlight36 citations
3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation
Peter Richtarik, Igor Sokolov, Elnur Gasanov, Ilyas Fatkhullin, Zhize Li, Eduard Gorbunov
Abstract
We propose and study a new class of gradient compressors for communication-efficient training—three point compressors (3PC)—as well as efficient distributed nonconvex optimization algorithms that can take advantage of them. Unlike most established approaches, which rely on a static compressor choice (e.g., TopK), our class allows the compressors to
BibTeX
@InProceedings{pmlr-v162-richtarik22a,
title = {3{PC}: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation},
author = {Richtarik, Peter and Sokolov, Igor and Gasanov, Elnur and Fatkhullin, Ilyas and Li, Zhize and Gorbunov, Eduard},
booktitle = {Proceedings of the 39th International Conference on Machine Learning},
pages = {18596--18648},
year = {2022},
editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
volume = {162},
series = {Proceedings of Machine Learning Research},
month = {17--23 Jul},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v162/richtarik22a/richtarik22a.pdf},
url = {https://proceedings.mlr.press/v162/richtarik22a.html},
abstract = {We propose and study a new class of gradient compressors for communication-efficient training—three point compressors (3PC)—as well as efficient distributed nonconvex optimization algorithms that can take advantage of them. Unlike most established approaches, which rely on a static compressor choice (e.g., TopK), our class allows the compressors to