Layer-wise Deep Neural Network Pruning via Iteratively Reweighted Optimization

Abstract

The huge number of parameters of deep neural network makes it difficult to deploy on embedded devices with limited hardware, computation, storage and energy resources. In this paper, we shall propose a log-sum minimization approach to prune a trained network layer by layer thereby improving the network compression ratio. Specifically, this is achieved by enhancing sparsity for network parameters such that the output of the network after pruning is consistent with the original one. We further present an iteratively reweighted algorithm to solve the nonconvex and nonsmooth log-sum minimization problem with general convex constraints. Furthermore, we show the existence of the cluster points for the iterates and the global convergence of the proposed iteratively reweighted algorithm. Numerical experiments demonstrate that the proposed approach is able to significantly prune the trained neural network while preserving the prediction accuracy.