Diverge to Converge: Mutual Heterogeneous Learning for Robust Pruning

Abstract

Neural network pruning is crucial for efficient deployment on resource-constrained devices, yet achieving high sparsity often leads to significant robustness degradation against adversarial perturbations and corruptions. Recent works typically rely on single-model fine-tuning along a fixed optimization trajectory, which renders the network susceptible to local optima and noise while failing to restore the multiple robustness properties compromised during compression. In this paper, we propose \textbf{Mutual Heterogeneous Learning (MHL)}, a framework enabling robust pruning via \textbf{single-model inference}. \textbf{MHL} instantiates heterogeneity through two complementary mechanisms: \textbf{layer-wise Lipschitz regularization} for intermediate feature smoothness and \textbf{adaptive margin objective} for difficulty-aware boundary separation. To guide these diverse experts to converge, we employ \textbf{entropy-based mutual distillation with a strategic schedule} that shifts the optimization trajectory from exploring diverse feature subspaces to consolidating a unified robust model. Extensive experiments on 4 clean and corruption benchmarks and adversarial attacks demonstrate that MHL significantly outperforms single-model baselines in both adversarial robustness (+5\%) and corruption robustness (+2.6\%) while maintaining competitive clean accuracy.