Toolkit enables seamless blending of real and virtual worlds using 3D splatting
SplatXR: A Toolkit for Multi-Layered Extended Reality via On-Headset 3D Gaussian Splatting
Human-Computer Interaction
Summary
Combining real and virtual worlds in extended reality (XR) is hard because traditional methods treat them separately. The authors created SplatXR, a toolkit that uses 3D Gaussian splatting to blend physical and digital objects together smoothly on XR headsets. This approach allows digital and physical items to interact naturally, like casting shadows or colliding, which was difficult before. Their toolkit also includes ways to capture and reconstruct objects on the headset itself, making mixed reality experiences more immersive and interactive.
What this means in practice
- •For xr application developers: Create immersive apps that integrate real and virtual objects with natural interactions using 3D Gaussian splatting on XR headsets.
- •For game developers: Build games where physical objects blend into virtual scenes with realistic physics and lighting effects using the splat-based toolkit.
Authors
Cyrus Vachha, Mohamed Kari, Parastoo Abtahi
Abstract
Recent advances in Gaussian splatting, 3D foundation models, and video-passthrough mixed reality offer unprecedented fidelity for capturing and blending physical and virtual worlds. However, established XR methods rely on polygon-based pipelines that treat physical and digital environments as separate domains, making seamless integration across recent novel representations difficult. Because splats lack explicit surface geometry, they impede standard graphics techniques necessary for cross-reality interactions, such as shadow or collision handling. Furthermore, while 3D foundation models can reconstruct physical objects on-the-fly into digital splats, current XR systems fall short of leveraging this capability for user experiences that transition seamlessly between physicality and virtuality. To address this, we organize XR as a multi-layered reality of interacting primitives, letting pixels and objects move across physical and digital boundaries. We contribute SplatXR, a Unity toolkit that operationalizes this structure as building blocks for interactive, immersive apps leveraging 3D Gaussian splatting: a standalone renderer, splat-to-mesh reconstruction, splat physics, splat blending, and offloaded splat object reconstruction via on-headset capture. SplatXR unites vision and graphics techniques to make splat-based XR development broadly available, and we validate it through demonstrative applications and a small developer study.