Papers for

virtual production teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Physically realistic dynamic 3D scenes generated from text prompts

ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptions

Abstract: Recent progress in image and 3D scene generation has enabled increasingly realistic static environments, yet most methods remain confined to such static configurations. Generating dynamic scenes from natural language is fundamentally challenging: it requires joint reasoning over scene structure, temporal evolution, and physical feasibility, while ensuring reliable execution in modern physics engines. We present a unified framework for generating physically consistent dynamic 3D scenes from text, with outputs directly executable in Unreal Engine. Central to our approach is the \emph{Evolutive Scene Graph} (ESG), which specifies entities with physical attributes, spatial relations, and event-driven timelines in a machine-checkable form. Given a prompt, a large language model constructs and validates a complete ESG; spatial layouts are grounded via energy-minimized gradient optimization; timeline-constrained physical parameters are then optimized through differentiable simulation to satisfy user-specified events; and the resulting scene is compiled into an engine-executable class. Experiments on 10 scenes across three complexity levels show that our method achieves $16.4/18$ mean event completion, outperforming Scene Language, the strongest engine-executable baseline (SimWorld), and our ablation without physical optimization by a clear margin in event completion and parameter accuracy.

Mon 14 SeptGraphicsComputer Vision and Pattern Recognition
The gist
Creating moving 3D scenes from text descriptions is tricky because the scenes need to make sense physically and change over time correctly. The authors designed a new way to build detailed 3D scenes that follow physical rules and happen over time, starting from simple text. Their system uses smart language models to plan the scene and physics simulations to make sure everything behaves realistically when run in game engines like Unreal Engine. This means you can describe a scene with events, and get a reliable, playable 3D scene that matches the text.
Open 2609.15392v1