AI summaryⓘ
The authors developed a new method called FF-ProCams that quickly creates detailed 3D models from projector-camera systems, which shine patterns of light on objects to understand their shape and appearance. Unlike older methods that take a long time to adjust for each scene, their approach uses a smart neural network to process a few pictures all at once and produce a model that can be relit by different projector settings. They trained their system on a large simulated dataset and showed it works well even on new objects and lighting, doing so much faster than traditional techniques. This can help improve augmented reality and projection mapping applications by making 3D reconstruction both accurate and efficient.
Projector-camera systemInverse rendering3D reconstructionFeed-forward networkGaussian representationDifferentiable rendererMulti-view observationSpatial augmented realityProjection mappingSurface reflectance
Authors
Ziyao Wang, Yuqi Li, Wenxing Zheng, Jiaying Chen, Chong Wang
Abstract
Projector-camera (ProCams) systems achieve active scene perception and controllable appearance manipulation via structured illumination, serving as a core infrastructure for spatial augmented reality, projection mapping, and surface reflectance acquisition. Existing inverse-rendering methods for ProCams deliver high-fidelity results but rely on time-consuming per-scene optimization, while mainstream feed-forward 3D reconstruction models produce baked appearance that cannot adapt to spatially varying projector illumination. To resolve this accuracy-efficiency trade-off, we propose FF-ProCams, a Feed-Forward 3D Gaussian inverse-rendering framework for ProCams. A hybrid Mamba2-Transformer encoder aggregates cross-view geometric and photometric cues from sparse multi-view observations, and lightweight heads predict a relightable Gaussian representation in a single forward pass. We further design a projector-aware differentiable renderer to synthesize camera observations under arbitrary active illumination and ProCams poses. To enable feed-forward training, we construct a large-scale synthetic ProCams dataset covering diverse object geometries and surface materials. Experiments show FF-ProCams achieves high-fidelity projector-aware rendering, generalizes to unseen patterns, and supports novel projector-camera poses. Using only 8 input views, it outperforms optimization-based baselines with 297 views while reducing test-time reconstruction to 0.13 seconds (a three-to-five-order-of-magnitude speedup). The code and data are available at https://github.com/CPREgroup/FF-ProCams/.