Selectively Extracting and Injecting Visual Attributes into Text-to-Image Models

Abstract

Text-to-image models are increasingly utilized in design workflows, but articulating nuanced design intentions solely through text remains a challenge. This work proposes a method that extracts visual attributes from a reference image and injects them directly into the generation pipeline. Specifically, the method optimizes a text token to selectively represent the target attribute using a custom training prompt and two novel embeddings, termed distilled and residual. A wide range of attributes can be extracted through this approach, including shape, material, pose, and camera shot and angle. The effectiveness of the method is validated on various target attributes and text prompts drawn from a newly constructed dataset. The method outperforms existing approaches in selectively extracting and applying target attributes across diverse contexts. Ultimately, the proposed method enables intuitive and controllable text-to-image generation, streamlining the design process.