Distributed Large Neural Network with Centralized Equivalence

Xinyue Liang; Alireza M. Javid; Mikael Skoglund; Saikat Chatterjee

ICASSP 2018accepted0 citations

Distributed Large Neural Network with Centralized Equivalence

Xinyue Liang, Alireza M. Javid, Mikael Skoglund, Saikat Chatterjee

Abstract

In this article, we develop a distributed algorithm for learning a large neural network that is deep and wide. We consider a scenario where the training dataset is not available in a single processing node, but distributed among several nodes. We show that a recently proposed large neural network architecture called progressive learning network (PLN) can be trained in a distributed setup with centralized equivalence. That means we would get the same result if the data be available in a single node. Using a distributed convex optimization method called alternating-direction-method-of-multipliers (ADMM), we perform training of PLN in the distributed setup.

BibTeX

@inproceedings{icassp2018_distributedlarge,
  title = {Distributed Large Neural Network with Centralized Equivalence},
  author = {Xinyue Liang and Alireza M. Javid and Mikael Skoglund and Saikat Chatterjee},
  booktitle = {ICASSP 2018},
  year = {2018}
}