PLoRA: Efficient Concurrent LoRA Training for Large Language Models
Minghao Yan, Zhuang Wang, Zhen Jia, Shivaram Venkataraman, Yida Wang
Abstract
Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated improving LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.
BibTeX
@inproceedings{
yan2026plora,
title={{PL}o{RA}: Efficient Concurrent Lo{RA} Training for Large Language Models},
author={Minghao Yan and Zhuang Wang and Zhen Jia and Shivaram Venkataraman and Yida Wang},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=khYw62jVZ2}
}