2025
Differentially Private Learning Needs Better Model Initialization and Self-Distillation
NAACL 2025long
Differentially private SGD (DPSGD) enables privacy-preserving training of language models, but often reduces utility, diversity, and linguistic quality. We introduce DPRefine, a three-phase method that initializes a model using data synthesis from a small pre-trained LM with rigorous filtering, appl…