2026
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
ICML 2026poster
Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show—both theoretically and empirically—that these pairs exhibit signific…