A Prior Driven Semi-Supervised ViTGAN for Image Recolorization

Abstract

This paper proposes a prior driven semi-supervised ViT-GAN called RC-ViTGAN for recoloring images while retaining color harmonization and semantic rationality. The encoder of RC-ViTGAN is based on the vision transformer to avoid the locality of convolutional networks, which facilitates the extraction of global information from images. Besides, we release an RC500 dataset, which is the largest publicly accessible and pioneering dataset for recolorization, providing convenience for subsequent studies. In addition, we present a novel semi-supervised training strategy, including a prior-driven self-supervised initialization method using contrastive learning. The proposed training strategy leverages massive amounts of unlabeled and pseudo-labeled data, addressing the shortage of labeled data in re-colorization. Code and dataset are available at https://github.com/tsz12/RC-ViTGAN.git.