Vision language model speeds up satellite image queries using fewer tokens
Exploiting answer-invariant redundancies in satellite imagery for efficient VLM inference on edge
Computer Vision and Pattern RecognitionMachine Learning
Summary
Satellites can use vision-language models to answer questions about images they take, but looking at every small part (tile) of a large image takes a lot of time and power. The authors identify parts of the image and its analysis that don't change the answer and can be skipped. They created a system called Rift that removes unnecessary parts before and during processing to save energy and speed up responses. Rift was tested on a smaller model running on an edge device and achieved big improvements in speed, energy use, and accuracy.
What this means in practice
- •For satellite system engineers: Run onboard vision-language queries more quickly and with less power by skipping image parts that don’t affect answers.
- •For edge ai developers: Build energy-efficient vision-language inference pipelines on resource-limited devices by dynamically pruning irrelevant image tiles and tokens.
Authors
Ishani Janveja, Davis Zhang, Seoyul Oh, Deepak Vasisht
Abstract
Onboard vision-language models could enable satellites to answer queries directly, but exhaustive tiled inference over high-resolution imagery is slow and energy-intensive. We identify answer-invariant token redundancy (AITR): image tiles and vision tokens that can be removed without changing the final answer. We present Rift, a two-stage system that performs query-conditioned tile pruning followed by elastic prefill to reduce token budget. We evaluate it on LLaVA-1.5 7B running on Jetson AGX Orin. Compared with exhaustive tiled inference, Rift reduces energy by 78% and latency by 69%, while increasing accuracy from 45% to 73%.