Scale does not improve fault tolerance in tool-calling LLM agents

Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance

Software EngineeringArtificial Intelligence

Summary

Large language model (LLM) agents use external tools to complete tasks, but usually these tools are assumed to work perfectly. The authors studied what happens when these tools fail in realistic ways, like responding late or incorrectly. They built MCP-Injector to simulate such faults based on real issues from many open-source projects. Their experiments show that bigger model size does not make agents more resilient to these tool failures, but the software frameworks handling the tools do matter. This means testing LLM agents only on ideal tool behavior can hide serious weaknesses.

What this means in practice

  • For ai system developers: Improve error handling in tool-calling LLM agent frameworks to increase robustness against real-world tool failures.
  • For software reliability engineers: Use protocol-aware fault injection like MCP-Injector to test and enhance operational fault tolerance of AI tool integrations.

Authors

Antonio Ken Iannillo, Joshua Segun Owotogbe, Roberto Natella, Francesco Avallone, Kristina Kudryavtseva, Indika Kumara

Abstract

LLM agents increasingly rely on external tools exposed via the Model Context Protocol (MCP), yet current benchmarks assume tools respond correctly and on time, leaving agent resilience to realistic tool failures untested. We introduce MCP-Injector, a drop-in, protocol-aware fault-injection middleware that interposes on the MCP tool-call path with controlled persistence (permanent, transient, intermittent) and seedable schedules. Its fault model is calibrated to the observed incident distribution (failure mode x persistence) of the public MCP ecosystem, mined from 57,240 issue and pull-request threads across 1,774 open-source MCP repositories. In a 1,350-cell experiment crossing two backbone sizes (20B, 120B), three agent stacks (custom multi-round plan-and-execute, OpenAI Agents SDK, LangChain), and three fault injection conditions, we find that faults drawn from this permanent-heavy stress profile cause severe quality degradation (task-completion score falls by 60%, from 8.08 to 3.23 out of 10), and that scale does not purchase fault tolerance for the two gpt-oss backbones we test: the 120B retains no larger share of its fault-free score under faults (resilience-ratio model, $p = 0.96$), and its completion rate under faults is, if anything, lower (91.1% vs. 95.6%; difference 4.4 points, 95% CI $[-0.2, 9.3]$). Agent stacks differ in how well they absorb faults, which suggests that framework-level error handling matters more than model scale. These results demonstrate that capability benchmarks alone can mask operational brittleness, motivating protocol-aware fault injection as a complementary evaluation dimension for tool-calling agents.