Radar builds detailed 3D scenes with language understanding
3D Occupancy-Language Fields from mmWave Radar
Robotics
Summary
Radar can usually tell where things are and recognize some categories, but it struggles to keep a full 3D map that understands new language questions. The authors created TextRadar, which uses radar data to build a 3D map that links space and language descriptions together. This map can answer new questions about the scene by connecting observations from different viewpoints. They tested their method across several buildings and found it works well for recognizing structures and human poses but less so for colors or printed text.
What this means in practice
- •For autonomous vehicle teams: Create 3D maps from radar that understand natural language queries about the environment, even in poor visibility conditions.
- •For security surveillance operators: Use radar to build persistent 3D representations that can be queried with language descriptions for monitoring complex indoor areas.
Authors
Haowen Lai, Zhiwei Zheng, Mingmin Zhao
Abstract
Radar systems can recover scene geometry and recognize predefined categories, but these outputs do not provide a persistent spatial representation for open-ended language queries. Such queries require linking observations from different viewpoints to the same physical locations. We present TextRadar, a framework that constructs a scene-level 3D occupancy-language field from panoramic radar observations. The field combines occupancy with scale-dependent language features at surface locations, allowing the reconstructed scene to be queried with new descriptions. A single-frame radar distillation model predicts aligned ranges and language features from each radar observation. Using radar-estimated poses, we register these predictions to reconstruct the field from radar alone in a new scene. Experiments on approximately 20k synchronized observations across seven buildings demonstrate scene-level natural-language querying and prompt-free semantic visualization. Analysis of camera-to-radar concept transferability shows stronger agreement for structures, materials, and human poses than for colors and printed text.