Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address the challenge of creating accurate, editable 3D models of indoor scenes for robot simulations. Instead of expecting perfect data at each step, their method, Lucida, processes realistic, imperfect inputs and gradually refines them. Lucida builds a scene graph from video, generates detailed 3D objects from multiple views, and uses a smart policy called GizmoAct to place each object precisely. Their approach leads to much better accuracy in detecting, positioning, and reconstructing objects compared to previous methods.
composable scene modeling3D object detectionpose estimationscene reconstructionmulti-view evidencescene graphrobot simulationvisual language model (VLM)GizmoActADD-SB metric
Authors
Minghan Qin, Yuang Wang, Xiuyu Yang, Yushi Long, Yujian Zhang, Ruihuan Wang, Kai Ye, Yangang Zhang, Hang Li
Abstract
Composable scene modeling aims to recover a real indoor scene as complete, editable object assets arranged as observed, giving robot simulation and embodied AI a simulation-ready replica of the real environment whose objects can be manipulated individually. Existing pipelines decompose the task into three steps---parse the observations into instances, generate an asset for each, and place each asset back---but every step presumes an input that a cluttered capture rarely provides: accurate instance geometry, unoccluded views, and assets that accurately match the observations. We propose Lucida, which keeps this order but redistributes the requirements, so each step consumes only what a real capture reliably provides and precision is reached at the end of the pipeline rather than demanded at its start. Lucida parses the video into a scene graph whose nodes carry per-instance multi-view evidence, generates a complete asset for each instance from its evidence, and places assets with GizmoAct, a VLM policy that casts placement as multi-turn GUI interaction, manipulating the object's gizmo in a closed loop and deciding itself when alignment is reached. Across scene-level 3D object detection, object pose estimation, and scene reconstruction, Lucida improves mAP over Boxer by 69% on R2S-Scene, raises ADD-SB@0.05 from 57.8% to 83.4% on CA-1M, and increases scene F-Score from 0.794 for SAM3D to 0.924.