Visual Prompt Tuning for Generative Transfer Learning
Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, Han Zhang, Yuan Hao, Irfan Essa, Lu Jiang
Abstract
Learning generative image models from various domains efficiently needs transferring knowledge from an image synthesis model trained on a large dataset. We present a recipe for learning vision transformers by generative knowledge transfer. We base our framework on generative vision transformers representing an image as a sequence of visual tokens with the autoregressive or non-autoregressive transformers. To adapt to a new domain, we employ prompt tuning, which prepends learnable tokens called prompts to the image token sequence and introduces a new prompt design for our task. We study on a variety of visual domains with varying amounts of training images. We show the effectiveness of knowledge transfer and a significantly better image generation quality. Code is available at https://github.com/google-research/generative_transfer.
BibTeX
@inproceedings{cvpr2023_visualprompttuni,
title = {Visual Prompt Tuning for Generative Transfer Learning},
author = {Kihyuk Sohn and Huiwen Chang and José Lezama and Luisa Polania and Han Zhang and Yuan Hao and Irfan Essa and Lu Jiang},
booktitle = {CVPR 2023},
year = {2023}
}