Tools linked to large language models often misuse input data causing security risks
ToFu: Exploiting and Fuzzing the LLM-Orchestrator-Tool Trust Boundary
Cryptography and Security
Summary
Large language models (LLMs) sometimes work with external software tools by sending commands through an orchestration layer. The paper shows that these tools often do not properly check if the information from the LLM is safe and correct. The authors built a system called ToFu that can automatically test these tools and find security problems by looking at their documentation and trying many inputs. They found hundreds of real bugs, showing it’s risky to trust LLM arguments without proper checks.
What this means in practice
- •For software security engineers: Automatically detect and fix input validation vulnerabilities in tools connected to LLMs, improving system safety.
- •For llm platform developers: Integrate automated fuzzing and semantic checks to ensure safer tool interactions within LLM orchestration frameworks.
Authors
Md Nazmul Islam Ananto, Pierre Prevost-Avenel, Colin Smith, Kyungtae Kim, Taegyu Kim, Arslan Khan
Abstract
Large language models (LLMs) are increasingly deployed as agents that interact with external software through tools, with an orchestration layer translating model outputs into structured tool invocations. This architecture creates an LLM-orchestrator-tool trust boundary where tool semantics may go unenforced and LLM-generated arguments may be incorrectly treated as trusted. We present ToFu, a tool-aware framework that automatically derives semantic security checks for LLM tools from their existing metadata and a CWE-guided sanitizer corpus. ToFu specializes these checks to individual tools and automatically generates fuzzing harnesses and semantic oracles using runtime introspection and outcome sanitization. We evaluated ToFu across 273 MCP servers containing 3405 tools and identified verified bugs corresponding to CWE weakness classes in 343 tools across 118 of the servers. Our findings show that real-world tools frequently fail to enforce their documented input semantics, making it unsafe to treat LLM-generated arguments as trusted.