Selecting optimal layer reduction factors for model reduction of deep neural networks

Abstract

Deep neural networks (DNN) achieve very good performance in many machine learning tasks, but are computationally very demanding. Hence, there is a growing interest on model reduction methods for DNN. Model reduction allows to reduce the number of computations needed to evaluate a trained DNN without a significant performance degradation. In this paper, we study layerwise reduction methods that reduce the number of computations in each layer independently. We consider the pruning and low-rank approximation method for model reduction. Up to now, often a constant reduction factor is used in all layers. In this paper, we show that a non-uniform allocation of reduction factors to different layers can greatly improve the performance of the reduced DNN. For this purpose, we select the optimal layer reduction factors in terms of an optimization problem. Experiments on three different benchmark datasets demonstrate the superior performance of our method.