Detecting poisoned text segments in multi-source inputs to large language models
Detecting and Localizing Segment-Level Poisoning in Multi-Source LLM-Agent Inputs
Cryptography and Security
Summary
Large language models (LLMs) sometimes use text from many sources to answer questions or generate content, but some sources might add harmful or misleading information called poisoned segments. This paper shows that these harmful parts cause specific changes inside the LLM’s internal processing patterns. By studying these patterns, the authors created ActProbe, a system that can spot and pinpoint which pieces of text are poisoned without changing the LLM itself. This approach works well on multiple models and types of attacks, helping keep LLM outputs trustworthy and safe.
What this means in practice
- •For ai platform operators: Detect and remove harmful poison segments in multi-source inputs to improve the reliability of large language model outputs.
- •For cybersecurity teams: Identify and localize malicious text injections in aggregated user reviews or documents feeding into AI agents for threat mitigation.
- •For e-discovery service providers: Improve document vetting by detecting subtle manipulations in multi-source inputs to AI-assisted legal review tools.$Commercial implications: Enables development of premium AI tools for legally compliant and secure document analysis by detecting poisoned input segments.
Authors
Xue Tan, Changhui Wang, Sanrui Yang, Hao Luan, Zhuyang Yu, Jin Wei, Ping Chen, Xiaoyan Sun, Jun Dai
Abstract
Modern large language model (LLM) agents often construct prompts by aggregating retrieved passages, user reviews, and documents from multiple external sources. This paradigm exposes them to segment-level poisoning attacks, in which an adversary controlling only a small subset of sources injects malicious content to manipulate model outputs. Existing defenses mainly rely on textual patterns, external embeddings, or auxiliary detectors and may therefore fail against fluent, semantically plausible poisoned segments. They also provide limited support for locating the responsible segments. We observe that successful corrupted-evidence and adversarial-instruction attacks induce structured shifts in the LLM's internal activations, forming a consistent activation-space pattern that we call the poison direction. Based on this observation, we propose ActProbe, an internal-state-based framework for detecting and localizing poisoned segments in multi-source LLM inputs. ActProbe projects MLP activations onto a learned poison direction and uses a lightweight linear SVM trained on a small calibration set to detect contaminated prompts. It then applies BinRoL, which combines recursive replacement ablation, Mahalanobis-distance-based branch pruning, and MAD-based robust leaf detection to locate poisoned segments. ActProbe requires no modification to the backend LLM and reduces localization overhead from O(n) exhaustive probing to O(k log n) forward passes. Across three datasets, two attacks, and four open-weight LLMs, ActProbe achieves a 0.01 false-positive rate, a 0.05 false-negative rate, 0.94 localization recall, and a 0.90 localization F1-score. It remains effective against defense-aware adaptive attacks and can protect black-box APIs through surrogate-based poisoned-segment removal.