Christopher M. Jermaine

ResIST: Layer-wise decomposition of ResNets for distributed training

We propose ResIST, a novel distributed training protocol for Residual Networks (ResNets). ResIST randomly decomposes a global ResNet into several shallow sub-ResNets that are trained independently in a distributed manner for several local iterations, before having their updates synchronized and aggr…