Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning
Xinlong Li, Weijieying Ren, Wei Qin, Lei Wang, Tianxiang Zhao, Richang Hong
Abstract
Existing continual learning works explored strategies like memory replay, regularization, and parameter isolation, but little analysis was conducted on the optimization behavior of LLMs’ continual fine-tuning. In this work, we investigate the geometric connections of different minima along the continual LLM fine-tuning trajectories. We validate this phenomenon on LLMs and propose a new method called Interpolation-based LoRA (I-LoRA). I-LoRA can strike a balance between plasticity and stability through parameter interpolation, which constructs a dual-memory experience replay framework based on LoRA. Experiments on eight domain-specific benchmarks demonstrate that I-LoRA consistently shows significant improvement over previous approaches with up to 11% performance gains. Our code is available at https://anonymous.4open.science/r/LLMCL-3823.
BibTeX
@inproceedings{icassp2025_analyzingandredu,
title = {Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning},
author = {Xinlong Li and Weijieying Ren and Wei Qin and Lei Wang and Tianxiang Zhao and Richang Hong},
booktitle = {ICASSP 2025},
year = {2025}
}