Image generation improved with multi reference diffusion conditioning
RefCompose: Multi-Reference Image Generation via LoRA-Conditioned Diffusion
Computer Vision and Pattern Recognition
Summary
Creating images that combine multiple detailed references like people, places, and objects is hard because existing tools struggle to keep all details accurate or become too slow as more references are added. The authors propose RefCompose, a new method that uses a fixed-size image grid to keep data manageable and separates where things go from how they look. This approach keeps memory use steady regardless of how many references there are and better maintains the exact appearance and layout of each reference. As a result, RefCompose makes it easier to create complex, realistic images with many elements while running efficiently.
What this means in practice
- •For visual effects artists: Compose multiple characters and set elements into a single scene with precise control over layout and appearance without memory scaling issues.
- •For game developers: Generate detailed multi-object scenes efficiently by conditioning on several references while preserving individual details and spatial relationships.
Authors
Sai Sri Teja Kuppa, Parth Shinde, Priyadharsan Balaji S, Jinka Harshavardhan, Sriprabha Ramanarayanan
Abstract
Filmmakers and visual artists routinely need to compose multiple references, actors, locations, props, cultural elements, into a single coherent shot, but existing tools either fail to scale past a handful of references or destroy fine grained subject identity in the process, since per reference tokenization scales memory linearly with reference count $N$ and generated content often departs from the given references rather than reproducing them. We propose \textbf{RefCompose}, a pixel space compositional conditioning framework that decouples \emph{where} things go from \emph{what} they look like, via a single fixed resolution reference canvas that keeps conditioning size constant regardless of reference count. Spatial layout is induced at inference time from a frozen diffusion transformer and extracted via Grounding DINO, requiring no LLM or dedicated layout model, while dual stream LoRA adapters inject a layout derived depth map and the encoded canvas through separate low rank streams, disentangling geometric scaffolding from localized appearance. On the Dense Layout protocol, RefCompose consistently outperforms layout based and state of the art multi reference baselines on color, texture, shape, spatial accuracy, and identity/content preservation at higher reference counts, all with constant inference memory, making it a practical building block for multi subject cinematic composition at production scale.