2021
Nondeterminism and Instability in Neural Network Optimization
ICML 2021spotlight
Nondeterminism in neural network optimization produces uncertainty in performance, making small improvements difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple model copies, doing so is time-consuming, costly, and harms reproducibility. In this wo…