iReWindColor: Vision Transformer with Residual Embedding and Window Encoder for Point-Interactive Image Colorization

Abstract

Point-interactive image colorization is intended to colorize a grayscale image by allowing the user to specify colors at specific locations. The colors provided by the user (user hints) are propagated appropriately throughout the image to obtain a reasonably colorized image with minimal user effort. However, existing approaches result in excessive color propagation and color bleeding beyond the boundaries of the region. To address this problem, we propose iReWindColor. iReWindColor achieves fine colorization by using smaller patches for Vision Transformer-based processing. Due to the increase in computational cost caused by reducing the patch size, the processing for the entire image is replaced by Window Encoder, which is the window-based processing. This allows fine colorization with small patches without increasing the computational cost. In addition, we propose Residual Embaedding to reduce colorization divergence between windows. Residual Embedding makes the network consider the entire image. Furthermore, for better utilizing user hints, we propose Encode Block that consists of both Self-Attention and CNNs. Adopting it helps to capture the interrelationships between image patches. Quantitative and qualitative results show that our approach outperforms existing methods for point-interactive image colorization and produces color images that accurately reflect user intent with less computation cost.