LoQT: Low-Rank Adapters for Quantized Pretraining
Sebastian Bugge Loeschcke, Mads Toftrup, Michael Kastoryano, Serge Belongie, Vésteinn Snæbjarnarson
Abstract
Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose Low-Rank Adapters for Quantized Training (LoQT), a method for efficiently training quantized models. LoQT uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models. We demonstrate this for language modeling and downstream task adaptation, finding that LoQT enables efficient training of models up to 7B parameters on a 24GB GPU. We also demonstrate the feasibility of training a 13B model using per-layer gradient updates on the same hardware.
BibTeX
@inproceedings{
loeschcke2024loqt,
title={Lo{QT}: Low-Rank Adapters for Quantized Pretraining},
author={Sebastian Bugge Loeschcke and Mads Toftrup and Michael Kastoryano and Serge Belongie and V{\'e}steinn Sn{\ae}bjarnarson},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=Pnv8C0bU9t}
}