Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model
Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, Chun Yuan
Abstract
The Segment-Anything Model (SAM) stands as a foundational framework for image segmentation. While it exhibits remarkable zero-shot generalization in typical scenarios, its advantage diminishes when applied to specialized domains like medical imagery and remote sensing. To address this limitation, this paper introduces Conv-LoRA, a simple yet effective parameter-efficient fine-tuning approach. By integrating ultra-lightweight convolutional parameters into Low-Rank Adaptation (LoRA), Conv-LoRA can inject image-related inductive biases into the plain ViT encoder, further reinforcing SAM’s local prior assumption. Notably, Conv-LoRA not only preserves SAM’s extensive segmentation knowledge but also revives its capacity of learning high-level image semantics, which is constrained by SAM’s foreground-background segmentation pretraining. Comprehensive experimentation across diverse benchmarks spanning multiple domains underscores Conv-LoRA’s superiority in adapting SAM to real-world semantic segmentation tasks.
BibTeX
@inproceedings{
zhong2024convolution,
title={Convolution Meets Lo{RA}: Parameter Efficient Finetuning for Segment Anything Model},
author={Zihan Zhong and Zhiqiang Tang and Tong He and Haoyang Fang and Chun Yuan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ezscMer8L0}
}