Sparse Transfer Learning Accelerates and Enhances Certified Robustness: A Comprehensive Study
Zhangheng Li, Tianlong Chen, Linyi Li, Bo Li, Zhangyang Wang
Abstract
Certified robustness is a critical measure for assessing the reliability of machine learning systems. Traditionally, the computational burden associated with certifying the robustness of machine learning models has posed a substantial challenge, particularly with the continuous expansion of model sizes. In this paper, we introduce an innovative approach to expedite the verification process for L2-norm certified robustness through sparse transfer learning. Our approach is both efficient and effective. It leverages verification results obtained from pre-training tasks and applies sparse updates to these results. To enhance performance, we incorporate dynamic sparse mask selection and introduce a novel stability-based regularizer called DiffStab. Empirical results demonstrate that our method accelerates the verification process for downstream tasks by as much as 70-80%, with only slight reductions in certified accuracy compared to dense parameter updates. We further validate that this performance improvement is even more pronounced in the few-shot transfer learning scenario.
BibTeX
@article{Li_Chen_Li_Li_Wang_2025, title={Sparse Transfer Learning Accelerates and Enhances Certified Robustness: A Comprehensive Study}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/32539}, DOI={10.1609/aaai.v39i5.32539}, abstractNote={Certified robustness is a critical measure for assessing the reliability of machine learning systems. Traditionally, the computational burden associated with certifying the robustness of machine learning models has posed a substantial challenge, particularly with the continuous expansion of model sizes. In this paper, we introduce an innovative approach to expedite the verification process for L2-norm certified robustness through sparse transfer learning. Our approach is both efficient and effective. It leverages verification results obtained from pre-training tasks and applies sparse updates to these results. To enhance performance, we incorporate dynamic sparse mask selection and introduce a novel stability-based regularizer called DiffStab. Empirical results demonstrate that our method accelerates the verification process for downstream tasks by as much as 70-80%, with only slight reductions in certified accuracy compared to dense parameter updates. We further validate that this performance improvement is even more pronounced in the few-shot transfer learning scenario.}, number={5}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Li, Zhangheng and Chen, Tianlong and Li, Linyi and Li, Bo and Wang, Zhangyang}, year={2025}, month={Apr.}, pages={5084-5091} }