Block Circulant Adapter for Large Language Models

Xinyu Ding; Meiqi Wang; Siyu Liao; Zhongfeng Wang

IJCAI 20250 citations

Block Circulant Adapter for Large Language Models

Xinyu Ding, Meiqi Wang, Siyu Liao, Zhongfeng Wang

Abstract

Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs. Experiments show that our method uses 14× less number of parameters than VeRA, 16× smaller than LoRA and 32× less FLOPs than FourierFT, while maintaining close or better task performance. Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.

BibTeX

@inproceedings{ijcai2025_blockcirculantad,
  title = {Block Circulant Adapter for Large Language Models},
  author = {Xinyu Ding and Meiqi Wang and Siyu Liao and Zhongfeng Wang},
  booktitle = {IJCAI 2025},
  year = {2025}
}