Fast Training of Large Kernel Models with Delayed Projections
Amirhesam Abedsoltan, Siyuan Ma, Parthe Pandit, Mikhail Belkin
Abstract
Classical kernel machines have historically faced significant challenges in scaling to large datasets and model sizes—a key ingredient that has driven the success of neural networks. In this paper, we present a new methodology for building kernel machines that can scale efficiently with both data size and model size. Our algorithm introduces delayed projections to Preconditioned Stochastic Gradient Descent (PSGD) allowing the training of much larger models than was previously feasible. We validate our algorithm, \EP4, across multiple datasets, demonstrating drastic training speedups without compromising the performance. Our implementation is publicly available at: https://github.com/EigenPro/EigenPro .
BibTeX
@inproceedings{
abedsoltan2025fast,
title={Fast Training of Large Kernel Models with Delayed Projections},
author={Amirhesam Abedsoltan and Siyuan Ma and Parthe Pandit and Mikhail Belkin},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=a7hHwWnZey}
}