DiffDesign: A diffusion model using garment Knowledge-Enhanced for Fashion Design Synthesis

Abstract

Despite the popularity of existing text-to-image models, their utilization in the field of garment design has received limited attention and has primarily focused on a small number of explorations into garment style migration. This represents a missed opportunity to fully harness the potential of these generative models. To address this gap, we introduce DiffDesign, a generative model aimed at aiding garment designers in creating garments that not only integrate expertise in garment design, comprehend garment attributes, and generate images from textual descriptions, but also leverage the entire capability of the generative model to materialize desired elements onto the garment. DiffDesign is founded on the WÜRSTCHEN architecture to enhance semantic alignment between textual descriptions and images. Our proposed Expert Garment Knowledge Enhancement Module aids in learning key attributes from text and identifying salient regions in images, thereby addressing the issue of garment attribute confusion and better fulfilling the needs of garment designers. Experimental results demonstrate that our approach effectively enhances semantic alignment in text-to-image models, aligns with design descriptions, and accurately generates garment images with the correct attributes, thereby facilitating multimodal garment design for garment designers.