Fine-Grained Private Knowledge Distillation

Yuntong Li; Shaowei Wang; Yingying Wang; Jin Li; Yuqiu Qian; Bangzhou Xin; Wei Yang

ICASSP 2023accepted0 citations

Fine-Grained Private Knowledge Distillation

Yuntong Li, Shaowei Wang, Yingying Wang, Jin Li, Yuqiu Qian, Bangzhou Xin, Wei Yang

Abstract

Knowledge distillation has emerged as a scalable and effective way for privacy-preserving machine learning. One remaining drawback is that it consumes privacy in a client-level manner. In order to attain fine-grained privacy accountant and improve utility, this work proposes a model-free reverse k-NN labeling method towards record-level private knowledge distillation, where each private record is employed for labeling at most k queries. Theoretically, we provide bounds of labeling error rate under the centralized/local model of differential privacy. Experimentally, we demonstrate that it achieves new state-of-the-art accuracy in MNIST/SVHN/CIFAR-10 dataset with one order of magnitude lower of privacy loss.

BibTeX

@inproceedings{icassp2023_finegrainedpriva,
  title = {Fine-Grained Private Knowledge Distillation},
  author = {Yuntong Li and Shaowei Wang and Yingying Wang and Jin Li and Yuqiu Qian and Bangzhou Xin and Wei Yang},
  booktitle = {ICASSP 2023},
  year = {2023}
}