ControlGS improves extended reality image quality by adapting rendering
ControlGS: Conditioning Neural Gaussians for Downstream-Processing-Aware XR Rendering
Computer Vision and Pattern RecognitionGraphics
Summary
Images created for virtual or augmented reality go through many changes before you actually see them, like adjustments from your device and display. The way these changes happen can vary a lot depending on things like where you are looking or how much power your device uses. The authors created ControlGS, a method that adjusts the image-making process by taking these changes into account ahead of time, making the final images look better to the user. Their experiments show that ControlGS works well across various setups with little extra cost.
What this means in practice
- •For xr device developers: Improve visual quality in XR systems by adapting rendering to real-time downstream processing and device conditions.$Commercial implications: Enables XR device makers to deliver consistently higher visual quality across varying hardware and runtime conditions, enhancing end-user experience.
- •For graphics engine developers: Create rendering engines that optimize images factoring in display and post-processing variations to better match final perceived quality.
Authors
Weikai Lin, Junjie Zhao, Carl Marshall, Sushant Kondguli, Yuhao Zhu
Abstract
Extended Reality (XR) users do not directly perceive the output of a rendering engine. Instead, rendered images pass through a post-processing pipeline and the physical display-optics path before reaching the eye. Critically, the exact downstream processing can vary significantly at run time, influenced by, for instance, camera pose and display power budget. Traditional 3DGS methods either implicitly assume that this downstream pipeline preserves image quality or cannot adapt to downstream processing changes. To bridge this gap, we present ControlGS, an XR Gaussian rendering pipeline that optimizes end-to-end visual quality. ControlGS models and integrates the entire downstream processing, between the rendering output and the human eye, into the optimization objective. To adapt to downstream processing at run time, ControlGS dynamically generates Gaussian primitives conditioned upon the downstream processing parameters. Experiments show that ControlGS consistently improves end-to-end post-optics XR quality across different neural Gaussian backbones and datasets, with minimal overhead. Code is available at https://horizon-lab.org/controlgs/.