WeaveAgent improves tool use for ultra-high-resolution satellite images
WeaveAgent: A Two-Stage Tool-Routing Agent for Ultra-High-Resolution Remote Sensing Imagery
Computer Vision and Pattern Recognition
Summary
Working with ultra-high-resolution satellite images is tough because it’s expensive to process every tiny detail and hard to pick the right tools automatically. The authors created WeaveAgent, a two-step system that first decides what tools to call and then runs them on detailed parts of the images only when needed. This approach helps the system better handle vague or broad questions about satellite images. Their tests show it routes tool calls much more reliably than before, though accuracy gains need more work. WeaveAgent’s design separates decision-making from detailed image processing to save resources and improve reliability.
What this means in practice
- •For remote sensing engineers: Improve automated tool selection for analyzing ultra-high-resolution satellite images with vague input queries.
- •For geospatial software developers: Build systems that combine language models and image processing by decoupling routing decisions from expensive visual perception.
Authors
Zhongyu Pang
Abstract
Problem. Ultra-high-resolution (UHR) remote sensing with vague user intents has two bottlenecks: visual tokens are expensive, and tool calling must be format-reliable (pretrained models emit zero tool calls zero-shot). Method. WeaveAgent, a two-stage tool-routing agent, decouples routing from visual perception. Stage A is routing-first: emission is trained, not elicited. Stage B executes conditionally: intrinsic queries enter visual answering (full-scene thumbnail; a WeaveEarth-style evidence board as an optional fixed-budget, approx. 5k-token compression interface); extrinsic queries execute tool call on original full-resolution imagery, answering from tool observations in a second, observation-masked round. Training: alignment SFT, then GRPO under reward R_WA2. Results. Alignment SFT lifts extrinsic routing from 0% to 80.75% (323/400); GRPO suppresses 9 intrinsic mis-emissions while tool selection is unchanged. The trained 2B system does not beat the zero-shot 8B baseline overall (0.263 vs. 0.250), a diagnostic contribution. Oracle attribution separates two repair ingredients: loading the observation into context lifts extrinsic answer accuracy from 0.025 to 0.425 under marker-free cross-mode returns, and the two-turn SFT stage adds a further +9.3 points to 0.518 at a small routing cost. A +/- image ablation shows emission suppression is visually grounded, and a query-register matrix shows LLM-rewritten queries cost trained checkpoints 2-11 points. Scope. All training and evaluation use the 5,000 / 3,273 / 1,000-record VagueUHR corpus (600 intrinsic + 400 tool-requiring; the base seeds synthesis and is not used for optimization). Single-pass evidence construction runs at 7.31 s per image on an RTX 4090. Code, data, and evaluation protocols will be released.