Boosting Pruned Networks with Linear Over-Parameterization

Abstract

Structured pruning is a popular technique for reducing the computational cost and memory footprint of neural networks by removing channels. It often leads to a decrease in network accuracy, which can be restored through fine-tuning. However, as the pruning ratio increases, it becomes progressively more difficult to restore the accuracy of the trimmed network. To overcome this challenge, we propose a method that linearly over-parameterizes the compact layers, increasing the number of learnable parameters, and re-parameterizes them back to the original layers after fine-tuning, therefore contributing to better accuracy restoration. Similarity-preserving knowledge distillation is exploited to maintain the feature extraction ability of the expanded layers. Our method can be easily integrated without requiring additional technology, ensuring compatibility and can effectively enhance the pruning process for existing pruning techniques. Our approach outperforms fine-tuning on CIFAR-10 and ImageNet under different pruning strategies, particularly when dealing with large pruning ratios.