DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models
Yanming Liu, Xinyue Peng, Yuwei Zhang, Xiaolan Ke, Songhang Deng, Jiannan Cao, Chen Ma, Mengchen Fu
Abstract
Large language models have repeatedly shown outstanding performance across diverse applications. However, deploying these models can inadvertently risk user privacy. The significant memory demands during training pose a major challenge in terms of resource consumption. This substantial size places a heavy load on memory resources, raising considerable practical concerns. In this paper, we introduce DP-MemArc, a novel training framework aimed at reducing the memory costs of large language models while emphasizing the protection of user data privacy. DP-MemArc incorporates side network or reversible network designs to support a variety of differential privacy memory-efficient fine-tuning schemes. Our approach not only achieves about 2.5 times in memory optimization but also ensures robust privacy protection, keeping user data secure and confidential. Extensive experiments have demonstrated that DP-MemArc effectively provides differential privacy-efficient fine-tuning across different task scenarios.
BibTeX
@article{Liu_Peng_Zhang_Ke_Deng_Cao_Ma_Fu_Zhang_Cheng_Wang_Yin_Du_2025, title={DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/34830}, DOI={10.1609/aaai.v39i25.34830}, abstractNote={Large language models have repeatedly shown outstanding performance across diverse applications. However, deploying these models can inadvertently risk user privacy. The significant memory demands during training pose a major challenge in terms of resource consumption. This substantial size places a heavy load on memory resources, raising considerable practical concerns. In this paper, we introduce DP-MemArc, a novel training framework aimed at reducing the memory costs of large language models while emphasizing the protection of user data privacy. DP-MemArc incorporates side network or reversible network designs to support a variety of differential privacy memory-efficient fine-tuning schemes. Our approach not only achieves about 2.5 times in memory optimization but also ensures robust privacy protection, keeping user data secure and confidential. Extensive experiments have demonstrated that DP-MemArc effectively provides differential privacy-efficient fine-tuning across different task scenarios.}, number={25}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Liu, Yanming and Peng, Xinyue and Zhang, Yuwei and Ke, Xiaolan and Deng, Songhang and Cao, Jiannan and Ma, Chen and Fu, Mengchen and Zhang, Xuhong and Cheng, Sheng and Wang, Xun and Yin, Jianwei and Du, Tianyu}, year={2025}, month={Apr.}, pages={26317-26325} }