2025
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
ICLR 2025oral
Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the updates depending on how the two LoRA factors are scaled or rotated. This deficiency leads to ineffic…