Physics guided visual prompts improve robot navigation around invisible hazards
Seeing the Invisible: Physics-Guided Visual Prompting for Temperature- and Radiation-Aware VLA Navigation
RoboticsArtificial IntelligenceMachine Learning
Summary
Robots often use cameras to see obstacles, but some dangers like radiation or high temperatures can't be seen by normal cameras. The authors created a way for robots to 'imagine' these invisible risks by adding virtual obstacles that appear in the robot’s camera view based on physics data. This helps the robot avoid these dangers without needing to retrain or change its existing vision system. Their method works well in simulations and real robots near actual radiation and heat sources.
What this means in practice
- •For industrial robot teams: Enhance robot navigation in facilities with invisible hazards by overlaying virtual obstacles to improve safety without retraining models.
- •For autonomous systems developers: Add physics-based visual prompts to existing navigation policies to handle new sensor inputs for risks like radiation or temperature spikes.
- •For security robotics vendors: Develop navigation systems for robots that safely avoid invisible threats and market them to industries with hazardous environments.$Commercial implications: Enables sale of safer robotic navigation products for hazardous facilities by integrating physics-guided visual prompts without system retraining.
Authors
Hojoon Son, Fan Zhang
Abstract
Vision-Language-Action (VLA) models have become a major paradigm for Vision-and-Language Navigation (VLN). However, in safety-critical facilities, invisible risks such as radiation or temperature spikes cannot be detected by an RGB camera, and handling each risk is expensive, requiring a new encoder, new data, and model retraining. We propose Physics-Guided Visual Prompting (PG-VP), a plug-and-play multimodal perception module that instead reuses what a frozen VLA model already does well: avoiding visible obstacles. Given a proximal radiation or thermal source, PG-VP performs a physics-guided risk assessment to determine the avoidance direction and overlays a corresponding virtual obstacle that moves across consecutive frames (Dynamic Visual Prompting). The navigation policy then naturally detours around this invisible hazard. The identical virtual obstacle is used regardless of hazard type, so the visual prompting pattern remains fixed as sensors are added. When no hazard is detected, nothing is rendered, and the policy behaves exactly as it would without PG-VP. We evaluate PG-VP on OmniNav using the val-unseen splits of R2R-CE and RxR-CE, where it guides the policy toward intended low-risk actions in 84.9% and 83.2% of cases, at a cost of 6.8 and 7.9 percentage points in navigation success rate. We further test it with distinct scenarios on a real robot in the presence of actual thermal and radiation sources, all without any retraining. The real test shows that PG-VP effectively avoids these invisible hazards, improving worst-10% average trajectory safety by 63.45% and 32.59% against thermal and radiation sources, respectively.