Efficient Object Placement Via LLM and Diffusion Model

Abstract

We address the problem of object placement with user instructions using LLM and diffusion model. Traditional methods struggle to find a suitable location for filling the object with a semantically reasonable size. In this work, we leverage the LLM to predict the coordinates of the added object with the help of user instruction. First, We extend the object placement benchmark OPA-INST for object placement image editing. Second, we propose a framework Inst-GEdit to predict the coordinates of added object and blend the object with background via diffusion model in a semantically consistent and natural manner. Inst-GEdit can accomplish the object placement editing task with the user instruction in a mask free and training free manner. We conduct the experiments and evaluate our method on the OPA and image composition benchmark. We evaluate the results in both subjective and quantitative way, which demonstrate the effectiveness of our proposed method.