Vision language guided aerial robots connect multiple tasks seamlessly
From Local Whole-Body VLA Behaviors to Scene-Scale Aerial Manipulation
Robotics
Summary
Controlling flying robots with arms to complete tasks across large scenes is hard because it needs lots of examples and precise timing. The authors created a way to train these robots in simulations without real-world demos, using synced videos and robot motions. They developed a method to better align commands with what the robot actually does, even when there are delays. Their system understands language goals by linking them to objects and connects smaller robot skills to work together over large areas. They tested their approach in simulations and on a real flying robot arm, showing it can handle multi-step missions reasonably well.
What this means in practice
- •For drone operators: Control flying robots with arms to complete complex tasks across connected areas using natural language commands.
- •For robotics engineers: Train aerial manipulation policies in simulation without physical hardware, reducing development cost and time for multi-site tasks.
Authors
Weixiang Guo, Rui Jin, Haotian Jin, Xinhang Xu, Ruiyang Liu, Haoran Zhao, Yi Wang, Weiqi Gai, Kun Cao, Lihua Xie
Abstract
Vision-language-action (VLA) models enable task-conditioned interaction, but extending them to scene-scale aerial manipulation remains challenging due to costly whole-body demonstrations, latency-induced action-state misalignment, and cross-site behavior composition. We present a unified framework for synthetic policy training and scene-scale execution on articulated uncrewed aerial manipulators (UAMs). A scene-reconfigurable pipeline synthesizes task-conditioned, kinodynamically feasible trajectories and synchronized multiview observations for VLA training without physical-platform demonstrations. Measured-progress-aligned realization (MPAR) aligns asynchronously returned action chunks with measured execution progress and realizes them as continuous, dynamically feasible trajectories. A relational Scene Graph grounds language goals to object instances and feasible interaction regions, while topology-guided transfer connects local behaviors across sites. Local VLA skills achieve 39/60 successes (65.0%) in simulation under oracle target and feasible-handoff conditions. Under 500-ms added latency, with and without a transient command-update stall, MPAR reduces median takeover phase error by 0.212 s over nominal-time alignment. The complete system completes 21/50 simulated multi-site missions (42.0%) and is further validated on a physical articulated UAM.