LTOS: Layout-controllable Text-object Synthesis via Adaptive Cross-attention Fusions

Abstract

Controllable text-to-image generation synthesizes visual text and objects in images with certain conditions. However, existing visual text rendering and layout-to-image generation tasks focus on single modality generation or rendering, leaving yet-to-be-bridged gaps between the approaches correspondingly designed for each of the tasks. In this paper, we introduce a unified task called layout-controllable text-object synthesis (LTOS), which merges text rendering and layout-based object generation. We construct a layout-aware text-object synthesis dataset for the LTOS task, containing well-aligned labels visual text, and object information. We also propose a layout-controllable text-object adaptive fusion (TOF) model, which generates images with legible visual text and plausible objects. Specifically, we construct a visual-text rendering to synthesize text and employ an object-layout control module to generate objects while integrating the two modules to harmoniously generate and integrate text and objects in images. To enhance the image-text integration, we propose a self-adaptive cross-attention fusion that forces the image generation to attend more to important text spans. Within such a fusion, we use a self-adaptive learnable factor to learn to control the influence of cross-attention outputs. Experiments show that our model excels the strong baselines and obtain SOTA in LTOS. We release our dataset and codes <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.