SceneAgent improves 3D captures for realistic interactive simulations
SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Turning detailed 3D and 4D scene captures into interactive simulations is tricky because it requires fixing errors and adding physical details. The authors built SceneAgent, a system that uses computerized agents to inspect and repair 3D scenes so they can be used in simulations. It works with different kinds of captures like photogrammetry and LiDAR and improves accuracy significantly on benchmark tests. SceneAgent also creates varied object versions and demonstrations to help train robots to interact with the scenes.
What this means in practice
- •For robotics developers: Generate accurately modeled simulation environments from complex 3D and 4D captured scenes to train robot manipulation policies.
- •For game developers: Create realistic interactive environments by converting real-world scanned data into physically simulated and articulated scene objects.
Authors
Luke Hollis, Tianxing Fan, Heng Yang
Abstract
Visual captures provide detailed scene appearance, but converting them into interactive simulation environments requires object reconstruction, physical modeling, and correction of intermediate errors. We present SceneAgent, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair. The pipeline processes 3D Gaussian Splatting (3DGS), photogrammetry, LiDAR captures, AI-generated 3DGS scenes, and 4D videos of dynamic scenes, and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation. To support policy learning, SceneAgent generates diverse object variants ("digital sisters") and scripted demonstrations. On a cluttered bin scene from GraspClutter6D, SceneAgent raises F1 at 1 cm for every coding agent with scorable outputs both with and without its skills, and its reconstructions are better than SimFoundry: all agent configurations exceed SimFoundry's F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%. We further demonstrate diverse simulated interactions and preliminary policy transfer to a real robot.