CReStyler: Text-Guided Single Image Style Transfer Method Based on CNN and Restormer

Abstract

Text-guided image style transfer methods have gradually become a research hotspot. However, existing text-guided style transfer method suffers from content information missing and artifacts in the generated stylized images. Therefore, we propose CReStyler, a text-guided image style method based on the dual-branch structure of CNN and Restormer. In this work, in the first branch, we introduce a novel convolutional structure called FcasNet, which composes Frequency-domain Channel Attention Mechanism (FcaNet) and Cross Convolutional Block Attention Module (CRCBAM). It can generate rough style images l <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fca</inf> according to the input target text. In the second branch, the use of pixel-based Restormer constrains the phenomenon of fake images and content information missing due to excessive convolution. It can generate stylized images l <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">res</inf> with complete content information based on the target text. Finally, we combine l <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fca</inf> and l <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">res</inf> through weighted fusion to obtain refined stylized images. During the training process, we utilize directional CLIP loss to constrain text-image alignment. Experimental results show that our method produces better results compared with existing methods such as CLIPStyler, LDAST, Text2LIVE, InstructPix2Pix.