Hyper-Refinement for Low-Rank Adaptation
Savas Özkan, Taha Ceritli, Jeongwon Min, Eunchung Noh, Jung Min Cho, Dookun Park, Mete Ozay
Abstract
Parameter-efficient fine-tuning (PEFT) is utilized to adapt large pre-trained machine learning (ML) models to new tasks using a small number of trainable parameters. In particular, Low-Rank Adaptation (LoRA) is one of the prominent PEFT methods. To this end, we introduce a novel method that exploits data to improve the accuracy of models fine-tuned by LoRA. Our method implements a hypernetwork that generates refinement parameters using data to update the low-rank parameters of LoRA. The experimental results validate that our method improves the accuracy of large language models (LLMs) on language understanding tasks by 3% on average compared to LoRA and its variants.
BibTeX
@inproceedings{icassp2025_hyperrefinementf,
title = {Hyper-Refinement for Low-Rank Adaptation},
author = {Savas Özkan and Taha Ceritli and Jeongwon Min and Eunchung Noh and Jung Min Cho and Dookun Park and Mete Ozay},
booktitle = {ICASSP 2025},
year = {2025}
}