2024
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization
ACL 2024findings
Low-rank adaptation (LoRA) achieves parameter efficient fine-tuning for large language models (LLMs) by decomposing the model weight update into a pair of low-rank projection matrices. Yet, the memory overhead restricts it to scale up when the model size increases. We propose Randomized LoRA (RLoRA)…