ProtoDiffusion speeds up layered graphic design generation and editing

ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images

Artificial Intelligence

Summary

Creating graphics often involves stacking layers of images and colors, which can get large and complex. The authors developed ProtoDiffusion, a new way to represent and generate these layers more efficiently by sharing some parts of the image and only separately handling the parts that change. This approach makes it faster to create, edit, and rearrange layers from images or text descriptions. It also figures out how many layers are needed and where they go without extra instructions, making design generation more flexible.

What this means in practice

  • For graphic design software developers: Integrate faster generation and editing of multi-layer graphic designs from images or text, reducing storage and compute costs through compact layer representations.$Commercial implications: Enables new software features that produce editable layered designs efficiently, appealing to creative professionals and enterprises.
  • For digital content producers: Create multi-layered graphic assets from images or text prompts with flexible layer counts and placement without manual input, speeding up creative workflows.

Authors

Adrià Ciurana Lanau, Harleen Hanspal, Lusine Abrahamyan, Emilio Tylson Baixauli, Yu Cao, Danai Triantafyllidou, Stefano Zorzi, Gustavo Zanoni Felipe, Peter Griger, Stefan Sietzen, Petru-Daniel Tudosiu

Abstract

Graphic designs are ordered compositions of RGBA layers whose separation enables selection, movement, and reuse. Existing layered models represent each layer with a full-canvas or region-sized spatial latent grid, so representation size grows with the cumulative spatial extent of the layers. We present ProtoDiffusion, a latent-diffusion framework built on ProtoVAE, a multi-layer variational autoencoder that encodes a design into one shared spatial prototype and one candidate vector per layer. Each candidate conditions a shared decoder to reconstruct its corresponding RGBA layer. For $N$ layers, the representation contains $HW+N$ tokens before backbone-specific packing, compared with $NHW$ for full-grid-per-layer representations. ProtoDiffusion jointly denoises the prototype and a fixed set of candidate slots, conditioned on an image or text, with bounding boxes as optional layout guidance. The model leaves unneeded slots empty and determines the number of layers and, in absence of boxes, their layout, requiring neither as input. The same architecture supports both image-to-design and text-to-design. ProtoVAE achieves high-fidelity reconstruction from its compact shared latent representation, reaching 40.43 dB PSNR on layer crops. On the LayerD evaluation split, image-to-design achieves RGB L1 scores of 0.040 with bounding boxes and 0.057 without, with box-guided alpha soft IoU (0.850) comparable to the strongest fine-tuned baseline (0.871). On merged images, text-to-design reaches 66.41 FID versus 86.84 for FLUX.2 [klein], achieves a 0.860 text-accuracy score, and runs $19.1\times$ faster than the same-backbone generation-then-decomposition cascade.