Fast 3D scene building from photos for games and apps
FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute
Computer Vision and Pattern RecognitionRobotics
Summary
Creating detailed 3D scenes from regular photos or videos is usually slow and complicated. The researchers developed FIRE3D, a system that quickly turns simple color images or casual videos into ready-to-use 3D models with separate objects you can move and interact with. Their method works without extra fine-tuning after seeing new images, making it much faster than previous techniques. FIRE3D also produces high-quality shapes, positions, and textures for objects, suitable for games and interactive applications within about a minute.
3D scene reconstructionRGB imageRGB-D data6-DoF poseobject bounding boxmesh generationtexture mappingfeed-forward neural networksimulation-ready assetsamodal completion
Authors
Hongchi Xia, Tianhang Cheng, Wei-Chiu Ma, Shenlong Wang
Abstract
We present FIRE3D, a unified framework that takes a single RGB image or casual RGB video and transforms it into simulation-ready 3D scene assets for games and interactive applications in under a minute. At the core of FIRE3D is a feed-forward, end-to-end network that predicts a compositional scene representation from posed RGB-D observations estimated from the RGB capture, including the 6-DoF pose, bounding box, mesh, and texture for every object. By modeling the scene as a collection of discrete entities, FIRE3D produces amodally complete and simulation-ready environments where objects are physically decoupled and ready for interaction. Our framework requires no test-time optimization, runs orders of magnitude faster than prior interaction-ready methods, and provides object-level completeness beyond existing feed-forward 3D approaches. We demonstrate competitive or state-of-the-art results across pose accuracy, geometry completeness, and texture quality across various datasets while being orders of magnitudes faster. Project page: https://xiahongchi.github.io/Fire3D/