Bingshuo Qian

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in performance on previous tasks. In this paper, we propose Controlled LoRA (CLoRA), a…