Yuji Byun

Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity

Low-rank adaptation (LoRA) offers an efficient alternative to full-weight adaptation in federated fine-tuning of language models, significantly reducing computational costs. By adjusting ranks for each client, federated LoRA enables flexible resource allocation. However, we observe that heterogeneou…