NeurIPS 2020poster30 citations
Predicting Training Time Without Training
Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto
Abstract
We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. To do so, we leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. This allows us to approximate the training loss and accuracy at any point during training by solving a low-dimensional Stochastic Differential Equation (SDE) in function space. Using this result, we are able to predict the time it takes for Stochastic Gradient Descent (SGD) to fine-tune a model to a given loss without having to perform any training.
BibTeX
@inproceedings{NEURIPS2020_440e7c3e,
author = {Zancato, Luca and Achille, Alessandro and Ravichandran, Avinash and Bhotika, Rahul and Soatto, Stefano},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {6136--6146},
publisher = {Curran Associates, Inc.},
title = {Predicting Training Time Without Training },
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/440e7c3eb9bbcd4c33c3535354a51605-Paper.pdf},
volume = {33},
year = {2020}
}