Louis Hernandez

The impact of LoRA on Oversmoothing $\colon$ Understanding Catastrophic Forgetting in Mean-Field Attention Dynamics

Low-Rank Adaptation (LoRA) is the dominant parameter-efficient fine-tuning method due to its favorable compute-performance trade-off, yet it suffers from catastrophic forgetting. We study forgetting through a tractable _mean-field self-attention_ toy model, where tokens evolve as an interacting part…