Scarf accelerates 3d scene reconstruction with smarter depth and gaussian use
SCARF: A Scene-Adaptive Depth-Guided G-3DGS Encoder Accelerator with Semantic Reuse and Fused Dataflow
Hardware Architecture
Summary
Creating 3D scenes from multiple images can be slow and wasteful because existing methods search depths for every pixel and generate many unnecessary geometric shapes. The authors introduce SCARF, a special computer accelerator that speeds up this process by grouping similar pixels and skipping redundant calculations. It rethinks depth searching by focusing on similar features rather than scanning everywhere, and only creates necessary 3D shapes based on local scene regularities. This makes scene reconstruction faster and more energy efficient without visibly affecting image quality.
What this means in practice
- •For edge ai developers: Run faster and more energy-efficient 3D scene reconstruction on edge devices by reducing redundant depth searches and Gaussian computations.
- •For augmented reality engineers: Generate novel views of real-world scenes more efficiently by leveraging scene-adaptive dataflow to reduce computation and memory needs.
Authors
Zirui Ma, Zhihua Fan, Haibin Wu, Wenxing Li, Fulin Zhang, Dongrui Fan, Xiaochun Ye, Wenming Li
Abstract
Depth-guided, cost-volume-based Generalizable 3D Gaussian Splatting (G-3DGS) encoders reconstruct scenes as sets of Gaussian primitives directly from multi-view images without requiring per-scene optimization, thereby facilitating feed-forward novel-view synthesis on edge computing platforms. However, their shared encoder architecture presents two primary bottlenecks. First, the per-pixel depth search disregards semantic similarity within the 2D feature space, causing depth search operations to account for over 65% of memory traffic due to redundant exhaustive candidate scans. Second, the sequential pipeline of depth prediction followed by Gaussian generation neglects local 3D geometric continuity, resulting in more than 52% of generated Gaussians being redundant prior to instantiation. In this paper, we introduce SCARF, a dataflow-co-designed accelerator that aligns each bottleneck with a corresponding scene-level regularity. To mitigate the first bottleneck, Feature Similarity Depth Reuse reconceptualizes depth search as a problem of semantic locality: depth consistency is determined by affinity in feature space rather than physical address proximity. It transforms exhaustive per-pixel depth searches into anchor-centered local searches. To overcome the second bottleneck, Scene-Adaptive Early Sparsification disrupts the conventional serial depth-to-materialization boundary by probing each tile to infer local scene regularity. It then selectively activates only the minimally sufficient generation pathway, eliminating redundant Gaussian computations before primitive instantiation. On a 27.8 square millimeter 28 nm post-layout design, SCARF delivers a 2.94x geometric-mean end-to-end speedup with maximum PSNR degradation below 0.072%, 2.23 inferences per joule, and 34.5x GPU energy efficiency.