CTR-LORA: CURVATURE-AWARE AND TRUST-REGION GUIDED LOW-RANK ADAPTATION FOR LARGE LANGUAGE MODELS
Zhuxuanzi Wang, Xi Xiao, Chen Liu, Chenrui Ma, Yunbei Zhang, Xiao Wang, Smita Krishnaswamy, Tianyang Wang
Abstract
Parameter-efficient fine-tuning (PEFT) has become the standard approach for adapting large language models under limited compute and memory budgets. Although previous methods improve efficiency through low-rank updates, quantization, or heuristic budget reallocation, they often decouple the allocation of capacity from the way updates evolve during training. In this work, we introduce CTR-LoRA, a framework guided by curvature trust region that integrates rank scheduling with stability-aware optimization. CTR-LoRA allocates parameters based on marginal utility derived from lightweight second-order proxies and constrains updates using a Fisher/Hessian-metric trust region. Experiments on multiple open-source backbones (7B-13B), evaluated on both in-distribution and out-of-distribution benchmarks, show consistent improvements over strong PEFT baselines. In addition to increased accuracy, CTR-LoRA enhances training stability, reduces memory requirements, and achieves higher throughput, positioning it on the Pareto frontier of performance and efficiency. These results highlight a principled path toward more robust and deployable PEFT.
BibTeX
@inproceedings{icassp2026_ctrloracurvature,
title = {CTR-LORA: CURVATURE-AWARE AND TRUST-REGION GUIDED LOW-RANK ADAPTATION FOR LARGE LANGUAGE MODELS},
author = {Zhuxuanzi Wang and Xi Xiao and Chen Liu and Chenrui Ma and Yunbei Zhang and Xiao Wang and Smita Krishnaswamy and Tianyang Wang},
booktitle = {ICASSP 2026},
year = {2026}
}