Hanxu Hou

LoRAN: Improved Low-Rank Adaptation by a Non-Linear Transformation

In this paper, we study parameter-efficient fine-tuning methods for large pre-trained models. Specifically, we improve LoRA approaches to alleviate the performance loss from the constrained adapter by introducing a non-linear transformation (call it LoRAN). For a better adaptation, we also design a…