Efficient rendering speeds up view synthesis with baked scene info
TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
Computer Vision and Pattern Recognition
Summary
Creating new views of a scene from images usually requires complex and slow 3D processing. This paper introduces Test-Time Baking (TTB), a new way to speed up this process by teaching a small neural network to remember important information from multiple views. The authors show that their method makes rendering about 23 times faster without losing image quality. This balance between speed and quality is better than previous techniques that tried to speed up view synthesis.
What this means in practice
- •For game developers: Render new camera views in 3D scenes faster while keeping high visual quality in interactive applications.
- •For augmented reality developers: Generate realistic viewpoints quickly for AR experiences without the need for slow 3D geometry reconstruction.
Authors
Chin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu, Cheng Sun
Abstract
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io