GPPT: Gaussian Process-infused Prompt Tuning for Vision-language Models

Abstract

Pre-trained vision-language models (VLMs) have achieved remarkable success in image classification tasks, leveraging efficient prompt tuning methods. However, the reliability of fine-tuned VLMs in safety-critical scenarios remains a concern due to the under-explored issue of confidence calibration. To address this limitation, we introduce Gaussian Process-infused Prompt Tuning (GPPT), a novel framework that integrates a Gaussian process into the hidden representations of images. By utilizing random Fourier features (RFF) and Laplace approximation, GPPT enables end-to-end training and seamless integration with existing prompt tuning methods. Our extensive experiments on 11 diverse downstream datasets demonstrate that GPPT achieves competitive performance with deep ensembles in both prediction accuracy and calibration, while requiring only a fraction of the inference time.