Lightweight onboard llm improves safe drone navigation near people
Onboard Lightweight LLM Reasoning for Human-Aware Safe UAV Navigation
Robotics
Summary
Drones flying around people need to treat different objects like people, cars, and walls differently to stay safe. The authors show how to use a small language model running directly on the drone to understand what each obstacle is and decide how close it can safely get. Their method helps drones avoid collisions better than traditional methods that just see obstacles as simple shapes. This system runs fast enough to keep drones safe in busy urban areas without needing an internet connection.
What this means in practice
- •For drone developers: Build drones that intelligently adjust flight paths based on the type of nearby obstacles for safer urban operations.
- •For robotics engineers: Enable aerial robots to maintain context-specific safe distances when navigating around people and vehicles using onboard language reasoning.
Authors
Siwei Cai, Yuwei Wu, Vijay Kumar, Lifeng Zhou
Abstract
Aerial robots operating near people must treat a pedestrian, a parked car, and a wall differently, yet conventional perception--planning stacks reduce all of them to occupied geometry---a \emph{human-aware safety gap} in which category-specific clearance is unavailable to the planner and temporary occlusions remove pedestrian-specific constraints from its map. We present a modular, fully onboard pipeline that closes this gap by turning visual observations into context-dependent safety constraints for real-time UAV navigation. A shared BLIP encoder drives scene captioning and text-conditioned segmentation, which is fused with depth to localize obstacles in 3D. Obstacle-specific risk is inferred by a lightweight LLM (Qwen-2.5-0.5B) stabilized with retrieval-augmented generation and grammar-constrained decoding that restricts its output to a planner-compatible JSON schema. The LLM reasons once per object category rather than per frame: its verdicts are cached per class, so previously seen categories are resolved without inference and new ones trigger a fresh retrieval-grounded query; detections are additionally given persistent identities and held in the map across occlusions. The perception pipeline runs at 10\,Hz on a Jetson Orin NX, while fallback LLM queries take approximately 0.8\,s and remain off the control-critical path. In Gazebo, low-altitude urban navigation among pedestrians and vehicles shows a roughly $3\times$ reduction in collision events over a geometric planner, and semantics-aware landing among moving actors raises success from 34\% to 96\%. The 224-pixel encoder input limits reliable small-object perception at range; conservative pedestrian margins and short-term persistence mitigate this limitation during navigation.