Tool-interface granularity affects performance of local AI agents
MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
Summary
When AI agents use tools to help with tasks, how those tools are divided up matters a lot. The authors found that having a few well-sized tools works better than very small tools or one big tool. This improves how often the AI finishes tasks and gets its reasoning right, especially on small devices with less computing power. Interestingly, bigger AI models don't always do better if the tools are not split well. Their findings help design smarter toolkits for AI that run on limited hardware.
What this means in practice
- •For iot device developers: Design interfaces with a few well-sized tools to improve AI task success and efficiency on resource-limited devices.
- •For ai system integrators: Optimize tool decomposition for AI agents to balance reasoning quality and speed depending on model size and deployment constraints.
Authors
Demetris Paschalides, Moysis Symeonides, George Pallis, Marios D. Dikaiakos
Abstract
As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet underexplored factor. How funψtionality is decomposed into tools affects whether an agent can select the right tool and construct valid arguments. This choice is especially consequential at the edge, where resource constraints limit which models can run locally and scaling up is often not an option. We present MCP-GRANITE, an open-source extensible benchmark framework that treats tool-interface granularity as a controlled variable for MCP-based agents, evaluated under edge and IoT scenarios. It comprises 81 multi-step scenarios across 9 domains, instantiated at 4 granularity levels from fine-grained primitive tools to a single tool. We evaluate 9 locally deployed models (268M-20.9B parameters) across 8,748 trials using task completion, tool selection F1, argument accuracy, latency, and resource-usage metrics. Results show that a 4-tool interface offers the best trade-off, improving task completion by 16.4% over fine-grained primitives and 33.6% over a single monolithic tool, while nearly doubling argument accuracy. Model size is only weakly correlated with task completion and strongly with latency, while its association with argument accuracy is less robust, and a 3.2B model at the optimal granularity outperforms a 20.9B model at a mismatched one. These findings identify tool-interface granularity as a key design parameter for MCP-based agents.