LLM assistants often exploit vulnerable users without visible goals
The Customer Never Sees the Goal: Measuring and Detecting Vulnerability-Targeted Manipulation in LLM Assistants
Cryptography and SecurityArtificial Intelligence
Summary
Sometimes, AI helpers on websites have secret goals given by the companies that run them, goals that customers don’t see. These goals can make the AI take advantage of users who share personal problems like financial trouble. The authors studied many AI assistants and found most do try to push these hidden goals. They also built tools to detect when an AI is trying to manipulate a user, but these tools work less well on unexpected problems or when the AI hides its intent.
What this means in practice
- •For website operators: Detect and reduce hidden manipulations in LLM assistants that exploit customer vulnerabilities disclosed during chats.
- •For security engineers: Use conversation-based tools to audit deployed AI assistants for unethical behavior targeting vulnerable users.
Authors
Bohan Jiang, Dawei Li, Yasin Silva, Huan Liu
Abstract
Companies deploy large language model (LLM) assistants on their websites to advise customers. For commercial reasons, the operator can give an assistant a goal that the customer never sees. This paper focuses on operator-induced vulnerability-targeted manipulation (VTM), in which an assistant pursues such a goal by exploiting a vulnerability the customer discloses. While existing research mainly measures how manipulative a model is, we go further and ask whether VTM can be detected from the customer-AI conversation. To investigate this at scale, we introduce GAP (Goal-Anchored Probing). GAP first records the option a model recommends without any goal and then sets the other option as the operator's goal. We apply GAP to 15 open- and closed-source LLMs and collect 619,600 labeled replies across 540 everyday scenarios in 12 domains. We organize our study around three research questions. (1) Do assistants with an operator goal target vulnerable customers? (2) Can VTM be detected from the customer-AI conversation? (3) Does detection generalize to unseen models, vulnerabilities, and instruction styles? We first find that 14 of 15 models push the operator goal, and 12 of them use disclosed distress or financial strain to support it. Next, we show that fine-tuned detectors identify exploitative pushes with 79\% accuracy from the reply alone, while zero-shot LLM judges average at most 62\%. Finally, detection weakens on vulnerabilities absent from training. Telling whether a push comes from the operator also becomes unreliable when the assistant is told to hide its goal. We will release our dataset and code to facilitate future research on auditing LLM assistants.