RenderBench provides editable 3D twins for real video evaluation
RenderBench: Benchmarking Render-to-Real Video Transfer with Reconstructed Digital Twins
Computer Vision and Pattern Recognition
Summary
It's hard to check how well computer programs can turn simple 3D sketches into real-looking videos because you need matching real videos and perfect 3D models to compare against. The authors created RenderBench, a collection of 12 real-life scenes with detailed 3D twins and videos that show the same views and actions. This dataset lets people test how close the program's videos are to the real ones in appearance and motion. They carefully verified each scene so that it’s accurate and easy to edit for future tests.
What this means in practice
- •For video model developers: Evaluate how well video generation models reproduce real scenes using editable 3D twins and matching real videos.
- •For augmented reality teams: Improve AR scene understanding by testing how proxy renders match real-world videos in complex environments.
Authors
Dicong Qiu, Zhiyuan Xu, Yaosheng Liu, Feng Han, Bo Ye
Abstract
Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and motion. Evaluating this render-to-real capability requires a real target video depicting the same scene evolution, paired with an editable, geometrically registered 3D replica. Such data has traditionally required substantial manual modeling, calibration, and animation effort. We introduce RenderBench, a benchmark of 12 reconstructed real-world scenes spanning large-scale indoor environments and egocentric viewpoints, with both static and dynamic settings. Our construction pipeline combines visual geometry, neural reconstruction, and assisted 3D authoring. Each scene is decomposed into static objects and dynamic actors, registered to the capture cameras, and accepted only after multi-view geometric and temporal validation. Each evaluation unit contains appearance reference images, a held-out real target video, an editable digital twin, a matched proxy render, and renderer-native scene annotations. We evaluate transfer models against paired real target videos, retain PAI-Bench-C-compatible structural projections, and use scene annotations to localize failures by object, visibility, articulation, and motion. The first release retains 12 of 14 registered samples (85.7%), comprising 1,496 paired real-proxy frames. All released scenes pass file-integrity and environment-edit audits, while proxy diagnostics yield a depth si-RMSE of 0.2170 and instance mIoU of 0.3673. RenderBench provides paired real observations and editable scene state for assessing both appearance fidelity and preservation of geometry and dynamics.