Memory-poisoning defenses vary in hidden costs for harmless conversations
The Price of Safety: Benign-Case Utility and Token Overhead of Memory-Poisoning Defenses in LLM Agents
Cryptography and SecurityArtificial Intelligence
Summary
Memory-poisoning defenses help protect AI systems that use large language models (LLMs) from malicious attacks. This paper studied how these defenses affect normal, safe interactions where no attacks happen. They found that some defenses that check input or track data origin don’t reduce accuracy or add overhead in peaceful situations. However, a defense that reranks memories to filter threats sometimes removes correct information and adds extra cost, even when there is no attack. Overall, this shows that the way a defense integrates matters more for normal use than whether it relies on an AI model or not.
What this means in practice
- •For llm system engineers: Choose memory-poisoning defenses that minimize false quarantines and overhead during typical safe user interactions to improve system efficiency.
- •For cloud security teams: Balance security and performance by selecting defense layers that do not unjustly block legitimate data when protecting AI-driven services from memory poisoning.
Authors
Pritom Bhowmik
Abstract
Memory-poisoning defenses for LLM agents are typically evaluated by their ability to prevent attacks. However, the traffic they process is rarely adversarial. The cost of implementing a defense is paid with each interaction, while its benefits are only seen in a small percentage of cases. We developed a measurement setup that keeps the memory backend, retrieval process, and judge consistent across different conditions, changing only the defense itself. We test each condition three times across five conversations to distinguish the defense's real effects from noise inherent in the pipeline's runs, which remains significant even at temperature zero. Across three write-time defenses (input sanitization, provenance checking, and LLM-based anomaly detection) and one read-time defense (reranking), tested on entirely benign traffic, the write-time defenses show no utility cost we can resolve, with 95% confidence intervals spanning roughly +/-4.5 points and including zero. The reranker is different: it lowers core accuracy by 4.4 points (95% CI [-9.0,-0.05], bootstrap; McNemar p=0.064), a result that survives replication but sits at the edge of our resolution. Its clearer cost is mechanical rather than statistical. On conversations containing no attack, the reranker quarantines legitimate memories on 33.6% of adjudicated items, reaching as many as 106 false quarantines in a single conversation, at 2.7% token overhead. Stacking all four defenses does not compound this cost: the combined condition's accuracy loss is smaller, and its confidence interval includes zero, suggesting the write-time defenses may partly offset what the reranker discards. Where a defense intercepts the pipeline, not whether it uses an LLM, appears to determine its benign-case price.