CognitiveReality builds live semantic 3D maps for robot teleoperation
CognitiveReality: Robot-Agnostic Semantic Gaussian Mapping with an LLM Agent for Immersive Collaborative VR Teleoperation
Robotics
Summary
Teleoperators using robots can see where the robot is but often don’t know what objects are around or how well they’ve been seen. The authors created CognitiveReality, a system that turns a robot’s camera data into a smart 3D map showing what things are in the scene and how good that info is. This map can be viewed in virtual reality and used by a language-based robot assistant to help understand spoken commands and pointing actions. It works with different robot types and can keep track of objects even when the robot temporarily loses its location.
What this means in practice
- •For robot teleoperation teams: Create shared 3D semantic maps in VR that integrate robot data and language to improve remote robot control and understanding of scene objects.
- •For industrial inspection crews: Use immersive interfaces combined with object-aware mapping to direct robot navigation and re-observation tasks more accurately in complex environments.
Authors
Timofei Kozlov, Dmitrii Maliukov, Andrey Marchenko, Dmitrii Plotnikov, Miguel Altamirano Cabrera, Dzmitry Tsetserukou
Abstract
A photorealistic 3D view tells a teleoperator where a robot is, but not what the scene contains, how well each object has been observed, or how to turn pointing and speech into robot action. CognitiveReality turns a robot's RGB-D stream into a live, semantically indexed Gaussian-TSDF map shared by an operator in virtual reality and a tool-using language agent. One mapper binary serves any platform through configuration alone: it ingests poses from robot SLAM, joint kinematics, motion capture or an inline visual tracker, bridges localization outages with a shadow tracker and keyframe-anchored PnP, and maintains open-vocabulary instance identities with per-object quality at 2 Hz. Speech and controller rays are grounded against persistent scene objects through validated typed tools and operator-confirmed robot actions. In the controlled agent evaluation, the deployed local Qwen3-VL-8B router reaches 81.24\% tool exact match, while merge-aware replay correctly redirects 101 absorbed object identifiers. On robot data CognitiveReality exceeds a Gaussian-plus-SDF baseline by 2-8 dB; pose error through 5-40 s SLAM outages stays within 1-8 cm. Deployed live on two quadrupeds, the agent executed 26 of 30 navigation requests and 20 of 20 re-observation requests, raising object quality by 2-5 dB.