Short context changes can flip AI decision model answers
JevOut: Natural Context Can Flip Decision Models
Computation and Language
Summary
Decision models that pick answers based on language input are usually seen as reliable. But the authors found that adding just a few natural-sounding words to the input can change the model's answer from correct to a confident wrong choice. They tested this across multiple datasets and models, showing this behavior happens often. This means these systems can be fragile and their probability scores may not always be trustworthy for making real decisions.
What this means in practice
- •For chatbot developers: Detect when short context additions cause chatbots to change correct answers into wrong ones for safer conversational AI.
- •For customer support teams: Design checks to catch when background details in customer input might mislead automated decision tools into wrong outcomes.
Authors
Zixiang Xu
Abstract
Dedicated decision models such as Jev map unstructured language to probability distributions over finite choices, allowing their outputs to directly route requests, select tools, and trigger actions. Yet real-world inputs rarely arrive in isolation: they come with background details and surrounding context. We find that short additions that fit naturally into this context can nevertheless redirect an otherwise correct decision, even when the correct answer remains unchanged. To study this behavior, we fix a wrong target option for each initially correct item and use the model's option probabilities to refine fluent context additions while preserving the source, question, choices, and gold answer. Within 64 accepted target evaluations, the optimizer identifies contexts that redirect Jev on 312 of 508 initially correct decisions (61.4%); in 229 cases, Jev assigns at least 0.7 probability to the fixed wrong option. Across seven datasets, three additional decision systems show targeted flip rates of 64.9%-73.2% on decisions they initially answer correctly. Taken together, these results expose a pronounced fragility in current decision models: short, ordinary-looking context can shift a correct choice to a high-confidence wrong one. Because these models turn language directly into downstream choices, this sensitivity raises concerns about treating their probability outputs as reliable decision interfaces.