Shlomo Hoory

Learning and Evaluating a Differentially Private Pre-trained Language Model

Contextual language models have led to significantly better results, especially when pre-trained on the same data as the downstream task. While this additional pre-training usually improves performance, it can lead to information leakage and therefore risks the privacy of individuals mentioned in th…