Elevating Visual Prompting in Transfer Learning Via Pruned Model Ensembles: No Retrain, No Pain

Abstract

Visual Prompting (VP) has been gaining traction in the deep learning community, yet its performance often falls short when compared to traditional finetuning methods in transfer learning. In this study, we present a novel approach to enhance VP by leveraging the insights from the lottery ticket hypothesis. In contrast to the prevailing practice of retraining pruned models, our discovery reveals that merely pruning an initially pretrained model, without any subsequent retraining, can deliver a VP on par with its dense counterpart. Building upon this valuable insight, we present an ensemble strategy that leverages VP on pruned backbone models at different sparsity levels, aiming to enhance VP accuracy. To assess the effectiveness of our approach, we conduct extensive experiments across 4 model architectures and 12 diverse datasets. Our results consistently illustrate the potency of pruning ensembles in augmenting VP performance, with accuracy enhancements spanning from 1.3% to an impressive 12.96%. This not only narrows the gap with traditional finetuning methodologies but also establishes a new benchmark for VP techniques.