Large language models become more robust to input manipulation attacks
Evaluating and Improving the Robustness of Large Language Models to Input Sequence Variations
Cryptography and SecurityComputation and LanguageMachine Learning
Summary
Large language models can be tricked by small changes in the input, causing wrong or harmful responses. The authors develop new measurements and methods to test how stable these models are under such attacks and to improve their defenses. They created attacks that work well even when copied to different models and found ways to reduce these attacks' success rates significantly. Their methods include combining multiple models and adding protections for how tools are called in complex systems. They provide software tools to help detect and prevent these problems.
What this means in practice
- •For platform security teams: Test and improve defenses of large language models against input manipulation and prompt injection attacks in deployed systems.
- •For software engineers building multi-agent ai: Add authenticated tool-call verification and isolation patterns to reduce attacks on agentic systems integrating multiple AI tools.
- •For cybersecurity firms: Provide advanced attack and defense software suites to detect and mitigate trojan and input-based attacks on language models.$Commercial implications: Enables sale of LLM robustness testing and protection tools for enterprise AI security solutions.
Authors
Narek Maloyan
Abstract
Large language models (LLMs) in production systems face prompt injections, trojans (backdoors), and manipulation of automatic quality metrics. This thesis develops models, methods, and algorithms for evaluating and improving LLM robustness to adversarial input sequence variations. We propose R_stab(f), a generative robustness metric based on the Jensen-Shannon divergence between per-step output distributions under small input perturbations. For localized attacks we prove V(h) <= 1 - R_class(h), where R_class(h) is the probability that a decision operator h keeps its decision under small perturbations. For non-localized attacks we propose a calibrated empirical model. For LLM-as-a-Judge systems we develop ASA, an adaptive evolutionary black-box attack that reaches an attack success rate (ASR) of up to 73.8%, with transfer between open models up to 62.6%. On Trojan Detection Challenge 2023 data (Pythia-1.4B), surrogate triggers reach REASR ~0.99 while recall of the true triggers is ~0.17 against a baseline of ~0.14. On SaTML CTF 2024 we systematize four classes of bypasses of multi-layer defenses, which reduce the ASR from 90% to 15-25%. Committees of 5-7 heterogeneous models reduce the ASR for Gemma-3-4B by 47-55 percentage points, to 19.3% with 7 models. For agentic systems based on the Model Context Protocol (MCP), we propose AttestMCP, which attests tool calls with HMAC-protected packets at under 0.1 ms per call, and the Commit Boundary isolation pattern. On the MCPBench benchmark of 847 scenarios they reduce the average ASR from 53.7% to 12.4%. The methods are implemented in the JudgeGuard and TrojanArmor software suites and the MCPSec module.