LucidLyra makes 3D scenes from videos directly searchable with objects
LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation
Computer Vision and Pattern Recognition
Summary
Making detailed 3D scenes from videos usually needs lots of images or slow processing steps. The authors created LucidLyra, a system that builds 3D scenes with object details and categories directly from video data, without extra manual work or complicated setups. It uses knowledge from multiple AI models to understand scene shapes, colors, and objects together, allowing users to look around and find things in 3D worlds quickly. This can help robots or virtual tools work in these scenes more effectively.
What this means in practice
- •For robotics engineers: Build robots that understand and interact with newly seen environments by quickly creating 3D scenes with object identities from single videos.
- •For game developers: Create editable 3D game assets and environments from video input without manual 3D modeling or multi-view scans.$Commercial implications: Enables game studios to generate detailed, interactive 3D scenes rapidly from video footage, reducing asset creation costs.
Authors
Oliver Hahn, Rodrigo Marcuzzi, Sangeetha Grama Srinivasan, Alessandro Burzio, Sherwin Bahmani, Jiahui Huang, Stefan Roth, Shengyu Huang, Qunjie Zhou, Laura Leal-Taixé
Abstract
We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.