Object based scene representation improves text guided video prediction
SlotDiT: Object-Centric Representations for Diffusion Transformers
Computer Vision and Pattern RecognitionRobotics
Summary
Predicting how scenes change over time is hard because images often mix up different objects. The authors propose a new method, SlotDiT, that breaks images into separate parts for each object before predicting changes. This helps their model better understand and forecast scene movements from text instructions. The method matches or beats other ways to generate videos and works well for robotic tasks. Plus, it does all this more efficiently by focusing on objects instead of whole images.
What this means in practice
- •For robotics teams: Improve robot planning by predicting object movements in scenes more reliably with object-based video generation.
- •For computer vision developers: Build more efficient text-conditioned video generation tools that separate scenes into object parts for better quality and speed.
Authors
Gjergj Plepi, Sven Behnke
Abstract
Text-conditioned latent diffusion models perform strongly in video generation and are promising backbones for robotic applications. However, existing approaches rely on pixel-level or VAE-based latent representations that lack explicit semantic structure, leaving the impact of the representation space largely unexplored. Slot-based object-centric representations offer a structured alternative by decomposing scenes into object-level latents, or slots. While they have shown success in dynamics modeling and planning, they have not yet been explored for diffusion-based generative modeling. We introduce SlotDiT, a text-guided Diffusion Transformer (DiT) that operates in a slot-based latent space. Given a reference image and a language instruction, SlotDiT decomposes the scene into object-centric slots representing individual entities. Conditioned on the instruction and observed scene context, the model autoregressively denoises future slot trajectories to predict scene dynamics. To systematically investigate latent-space design for diffusion transformers, we compare slot-based representations against VAE-based and semantics-aligned alternatives within a unified DiT framework. Our experiments show that using slots as DiT latents yields competitive video generation quality while consistently improving task-completion rates across four robotic datasets. Furthermore, their compact representation provides a computationally efficient alternative to VAE-based and semantics-aligned latent spaces. Overall, our results demonstrate that object-centric structure is a powerful inductive bias for diffusion-based generative modeling in robotic environments. The project page is available at https://slot-dit.github.io/.