On-device multimodal agents run faster and use less memory with aligned pruning
Parallelism-Aligned Pruning and Round-Aware Context Reuse for On-Device Multimodal Agents
Artificial Intelligence
Summary
Multimodal agents that work on devices like phones or edge computers face problems because their model parts don’t split evenly across processor cores and their memory fills up over time with old visual data. The authors designed PARC, a method that prunes model attention parts to fit processor cores better and reorganizes memory caches to remove old images more effectively. This lets the agents process visual information quicker, handle more interaction rounds within fixed memory limits, and keep accuracy almost unchanged. Their experiments show faster execution and lower memory usage without losing performance.
What this means in practice
- •For edge device developers: Speed up and reduce memory use in on-device multimodal AI by aligning model pruning with hardware cores and optimizing cache management.
- •For mobile app developers: Enable apps to run multimodal agents that respond faster and handle longer interactions without increasing memory demands on smartphones.
Authors
Peipei Jiao, Mingkang Liu, Weiming Meng, Lichen Pang, Tianyuan Liu, Junyan Li
Abstract
On-device multimodal agents are limited by two deployment mismatches: pretrained attention structures may not divide evenly across the compute cores of an edge accelerator, causing irregular partitions; and standard prefix caching keeps historical visual tokens in the persistent KV cache, whose capacity grows with the number of rounds. We present PARC, which combines parallelism-aligned structured pruning with lifetime-aware KV-cache reorganization. At the model level, it uses first-order Taylor importance and grouped-query-attention-aware selection to make attention dimensions divisible by the target core count while preserving the query-to-key/value ratio and head dimension. At runtime, it separates the system prompt, persistent text history, and transient visual observations by lifetime, updating text history during tool execution so that historical images leave the persistent cache and more interaction rounds fit a fixed budget. On 2,062 GUI actions, the incremental interface shows no accuracy regression, and task-aware recovery training restores 98.7% of the action-type accuracy gap and the coordinate hit rate to the original level. Under seven-core deployment, hardware-aligned pruning reduces vision-encoder latency by 11.4%, improves decoding throughput by 23.8%, and lowers per-heap high-water memory by 12.7%; a parameter-matched control confirms that the speedup stems mainly from core alignment rather than parameter reduction.