Global Static Pruning via Adaptive Sample Complexity Awareness

Abstract

Dynamic pruning leverage the feature information of each input sample to dynamically adjust the network structure, generating multiple subnetworks suitable for different sample complexity. However, it inevitably introduces higher computational complexity and increased memory consumption. In addition, complex multi-stage pipelines are required to counteract the performance degradation caused by pruning. In this paper, a simple yet effective global static pruning method based on Adaptive Sample Complexity Awareness is proposed, called ASCA, which achieves model compression without pre-training and fine-tuning. Specifically, an adaptive sample complexity-aware static pruning method is proposed, which leverages task loss to guide the network in enhancing or suppressing the feature learning of samples with varying complexities. Then, a new mask binarization loss is proposed to automatically distinguish important and unimportant channels, avoiding the impact of hand-crafted thresholds on pruning performance. Extensive experiments demonstrate that ASCA outperforms state-of-the-art pruning methods on CIFAR-10 and ImageNet datasets.