Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?

Boris Knyazev; DOHA HWANG; Simon Lacoste-Julien

ICML 2023poster20 citations

Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?

Boris Knyazev, DOHA HWANG, Simon Lacoste-Julien

Abstract

Pretraining a neural network on a large dataset is becoming a cornerstone in machine learning that is within the reach of only a few communities with large-resources. We aim at an ambitious goal of democratizing pretraining. Towards that goal, we train and release a single neural network that can predict high quality ImageNet parameters of other neural networks. By using predicted parameters for initialization we are able to boost training of diverse ImageNet models available in PyTorch. When transferred to other datasets, models initialized with predicted parameters also converge faster and reach competitive final performance.

BibTeX

@inproceedings{icml2023_canwescaletransf,
  title = {Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?},
  author = {Boris Knyazev and DOHA HWANG and Simon Lacoste-Julien},
  booktitle = {ICML 2023},
  year = {2023}
}