Edge computer chips share work to speed up vision language tasks
Shared Context, Separate Engines: Streaming Vision--Language Inference on Edge SoCs
Machine Learning
Summary
Edge devices like smartphones and small computers have multiple processors for handling tasks, but often only one processor works on vision and language problems, leaving others idle. The authors show a way to let these processors share information and work together on tasks that take video frames and generate language answers. This approach splits the stored memory so any processor can pick up and continue the task without starting over. Their method runs much faster than using just one processor, making real-time understanding of video and language on small devices more efficient.
What this means in practice
- •For mobile device engineers: Increase throughput and reduce response times of vision-language features on multi-processor mobile chips by coordinating engines for concurrent stateful inference.
- •For embedded systems developers: Implement efficient streaming vision-language services on edge hardware by sharing context across heterogeneous accelerators to maximize compute utilization.
Authors
Yilong Li, Chengpo Yan, Yijing Zeng, Hanlei Wu, Mingjun Song, Suman Banerjee
Abstract
Edge SoCs pack CPUs, GPUs, and NPUs, yet vision--language model (VLM) runtimes keep generation on one engine and leave the others idle. Streaming VLMs expose further parallelism: new frames arrive while earlier queries are still being answered. Engines differ in compute and memory access and contend when they co-run, so no single placement of stages wins across SoCs and workloads. Backend-local KV state keeps other engines from continuing the work. \name coordinates heterogeneous accelerators to execute stateful VLM inference concurrently. It separates a stream's versioned context from each engine's KV state, so any engine can continue the stream by reusing, importing, or reconstructing that state. Stateful Streaming Generation ingests new frames on one engine while another answers, and Concurrent Sequence Generation serves different ready requests in parallel. State-aware dispatch sends each request to the engine expected to finish it first. Each answer reads a fixed, published version of its history and matches its serial reference. \name beats the strongest single-engine baselines on Apple M4 Max, Rockchip RK3588, and Qualcomm QCS6490, even when they reuse context. On two RK3588 video streams, it delivers $19.6\%$ more throughput than the best single engine with context reuse and $27.2\%$ more than parallel engines without shared state. Cross-engine KV import cuts first-token latency by a factor of $4.6$ over prefix recomputation. Stateful Streaming Generation delivers up to $2.48\times$ the throughput of serial execution, and state-aware dispatch cuts query completion time by up to $72.7\%$.