Debiased Fine-Tuning for Vision-Language Models by Prompt Regularization
Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, Hanwang Zhang
Abstract
We present a new paradigm for fine-tuning large-scale vision-language pre-trained models on downstream task, dubbed Prompt Regularization (ProReg). Different from traditional fine-tuning which easily overfits to the downstream task data, ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning. The motivation is: by prompting the large model “a photo of a [CLASS]”, the fill-in answer is only dependent on the pretraining encyclopedic knowledge while independent of the task data distribution, which is usually biased. Specifically, given a training sample prediction during fine-tuning, we first calculate its Kullback-Leibler loss of the prompt prediction and Cross-Entropy loss of the ground-truth label, and then combine them with a proposed sample-wise adaptive trade- off weight, which automatically adjusts the transfer between the pretrained and downstream domains. On various out-of-distribution benchmarks, we show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.
BibTeX
@article{Zhu_Niu_Lee_Hur_Zhang_2023, title={Debiased Fine-Tuning for Vision-Language Models by Prompt Regularization}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/25496}, DOI={10.1609/aaai.v37i3.25496}, abstractNote={We present a new paradigm for fine-tuning large-scale vision-language pre-trained models on downstream task, dubbed Prompt Regularization (ProReg). Different from traditional fine-tuning which easily overfits to the downstream task data, ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning. The motivation is: by prompting the large model “a photo of a [CLASS]”, the fill-in answer is only dependent on the pretraining encyclopedic knowledge while independent of the task data distribution, which is usually biased. Specifically, given a training sample prediction during fine-tuning, we first calculate its Kullback-Leibler loss of the prompt prediction and Cross-Entropy loss of the ground-truth label, and then combine them with a proposed sample-wise adaptive trade- off weight, which automatically adjusts the transfer between the pretrained and downstream domains. On various out-of-distribution benchmarks, we show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.}, number={3}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Zhu, Beier and Niu, Yulei and Lee, Saeil and Hur, Minhoe and Zhang, Hanwang}, year={2023}, month={Jun.}, pages={3834-3842} }