Bangguo Ye

TRAC: Tensor-Train based Across-layer Compression for Parameter-Efficient Fine-Tuning

Fine-tuning large pre-trained models under resource constraints remains challenging due to the massive number of parameters involved. Existing parameter-efficient tuning methods, such as low-rank adaptation (LoRA) and its variants, rely heavily on matrix factorization and often struggle in extremely…