EdgeAgent speeds up multi-agent AI tasks on CPU-GPU chips
EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures
Distributed, Parallel, and Cluster ComputingMultiagent Systems
Summary
Running multiple AI agents on devices like phones or laptops can slow down because they compete for memory and computing parts. The authors created EdgeAgent, a system that helps these AI agents work together efficiently by managing memory and scheduling work smartly on chips that combine CPU and GPU memory. This approach keeps all parts busy and speeds things up even when the AI agents are waiting on external tools. Tests on an Apple M4 chip showed EdgeAgent can make these tasks up to 77% faster.
What this means in practice
- •For mobile device developers: Increase speed and efficiency of running multiple AI assistants simultaneously on phones and tablets with integrated CPU-GPU chips.
- •For embedded system engineers: Optimize AI workloads on edge devices that have unified memory architectures to better handle complex, multi-agent AI tasks while maintaining privacy.
Authors
Yuhai Long, Yuanxin Wei, Kai Wu, Jinhui Wei, Dan Huang, Jiangsu Du
Abstract
Emerging multi-agent LLMs demand privacy-preserving edge deployment, yet current inference systems struggle with these collaborative workflows. Specifically, the memory-bound decode phase causes severe bus contention on unified memory architectures (UMA), paralyzing naive CPU-GPU co-execution. Furthermore, speculative decoding in multi-agent workloads faces extreme variance in drafting difficulty, alternating between complex reasoning and predictable structured generation. Compounded by frequent tool-induced stalls, this highly fragmented execution severely underutilizes hardware and defeats traditional static batching. We present EdgeAgent, a cross-layer inference system explicitly co-designed for edge UMA and multi-agent workloads. At the micro-architectural level, it bypasses rigid graph-compiler constraints to enable zero-copy UMA-aware tensor parallelism, utilizing asymmetric memory layouts to fully saturate both CPU and GPU compute units. At the scheduling level, it dynamically allocates draft budgets based on real-time sequence predictability to bound bandwidth waste. Concurrently, an asynchronous suspend-and-yield mechanism actively evicts stalled agents, ensuring continuous hardware saturation during unpredictable tool invocations. Extensive evaluations on an Apple M4 SoC demonstrate that the UMA-aware execution alone contributes a 1.29x speedup over batched speculative decoding. Adding the agent-aware scheduling lifts the full EdgeAgent system to a 1.77x speedup under extreme tool-use latencies.