Personal AI devices balance speed cost and privacy in agent tasks
AgBench: Agentic AI Benchmarks for Personal AI Devices
Artificial IntelligencePerformance
Summary
Agentic AI systems often use big language models hosted in the cloud, which can cost money and share private data. The authors created AgBench, a test suite to measure how personal AI devices perform agent tasks locally, in the cloud, or a mix of both. They found local-only AI is slower and less successful but avoids cloud costs and data exposure. Using both local and cloud resources can help but depends on how work is split. No single method is best, so choices should fit the device and task.
What this means in practice
- •For mobile device developers: Evaluate and optimize AI task execution across device and cloud to balance cost, speed, and privacy using AgBench.
- •For enterprise ai platform teams: Select deployment architectures for agentic AI on personal devices guided by AgBench insights on success and data exposure trade-offs.
Authors
Yizhou Han, Di Wu, Dhananjay Saikumar, Blesson Varghese
Abstract
Agentic AI systems increasingly rely on cloud-hosted large language models for planning, tool use, and iterative execution, raising concerns about API cost and data exposure. Advances in personal AI devices enable agents to execute locally, but limited resources on device may affect task success and performance. Existing benchmarks are inadequate for systematically characterizing these trade-offs across devices, workloads, and deployment architectures. We present AgBench, a benchmark suite and open artifacts for reproducible evaluation of agentic AI on personal devices. Using AgBench, we evaluate local, hybrid, and cloud execution across agentic workloads, examining task success, latency, cloud API cost, and data exposure. Our results, drawn from over 162.07 million data points, show that personal AI devices can complete many agent tasks locally, but local-only execution generally has lower task success and longer completion times than cloud-only execution, especially as concurrency increases. Local-only execution eliminates cloud model API costs and sensitive-information exposure to cloud agents. Hybrid execution can improve task success, but its cloud cost and data exposure depend on how agents divide work and share information. No single architecture performs best across task success, goodput, cloud cost, and data exposure; deployment choices should reflect the intended workload and device capabilities. AgBench is available at https://anonymous.4open.science/r/AgBench-2777.