Small language models show trade off between reliability and speed on edge devices
Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment
Computation and Language
Summary
Small language models running on simple computers like Raspberry Pis can help do tasks without internet or cloud help. The authors tested five small models to see how well they can follow instructions that need exact, machine-readable answers. One model called Qwen2.5-1.5B was the best but still only got about 75% correct results. They also found that almost none of the model outputs were perfectly formatted and needed fixing after. Running these models on everyday CPUs takes significant time and memory, so there's a balance between reliability and resource use when using them on small devices.
What this means in practice
- •For embedded software engineers: Use the CPU benchmarks and parseability scores to select small language models for edge devices needing reliable structured outputs for tool control.
- •For iot system developers: Design on-device language model agents that balance memory use and response accuracy for applications like local data processing without cloud reliance.
Authors
Abrar Shahriar Qurat-Ul-Ain Mastoi
Abstract
Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions.