ICML 2024oral21 citations
LoRA Training in the NTK Regime has No Spurious Local Minima
Uijeong Jang, Jason D. Lee, Ernest K. Ryu
Abstract
Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models (LLM), but our theoretical understanding of LoRA has been limited. In this work, we theoretically analyze LoRA fine-tuning in the neural tangent kernel (NTK) regime with $N$ data points, showing: (i) full fine-tuning (without LoRA) admits a low-rank solution of rank $r\lesssim \sqrt{N}$; (ii) using LoRA with rank $r\gtrsim \sqrt{N}$ eliminates spurious local minima, allowing gradient descent to find the low-rank solutions; (iii) the low-rank solution found using LoRA generalizes well.
BibTeX
@inproceedings{
jang2024lora,
title={Lo{RA} Training in the {NTK} Regime has No Spurious Local Minima},
author={Uijeong Jang and Jason D. Lee and Ernest K. Ryu},
booktitle={Forty-first International Conference on Machine Learning},
year={2024},
url={https://openreview.net/forum?id=s1sdx6vNsU}
}