Maxime Sylvestre

Implicit Bias in Noisy-SGD: With Applications to Differentially Private Training

Training Deep Neural Networks (DNNs) with small batches using Stochastic Gradient Descent (SGD) often results in superior test performance compared to larger batches. This implicit bias is attributed to the specific noise structure inherent to SGD. When ensuring Differential Privacy (DP) in DNNs’ tr…