Summary
When AI models decide on refund or claim requests using written policies, they often look at past similar cases. This paper studies how showing the AI the recorded decisions of past cases versus just the facts of those cases influences its approval decisions in opposite ways. The authors found that changing past decisions shown to the AI can strongly increase approval rates, while showing only the facts of past cases can make the AI approve less often. This suggests AI behavior depends differently on precedent decisions and factual similarities. The findings highlight the importance of testing these two influences separately when designing AI systems that use past case data.
What this means in practice
- •For customer service teams: Improve AI decision-making in refund and claim handling by separately testing influences of past case decisions and case facts.
- •For legal tech developers: Design AI systems for legal or policy decisions that account for differing impacts of precedent labels and facts on model outputs.
Authors
Junwei Yao, Zike Yang, Yongxi Zhou, Yanxian Chen, Jianan Liu, Jiaxi Wen, Zhijun Wang
Abstract
When a language model decides refund or claim requests under written policies, the system can show it similar past cases. Each past case has two parts: its facts and the decision recorded for it. We separate the two. A model sees a policy, a request and three past cases, and we change either the recorded decisions or the facts, keeping the request fixed. We select requests that are close calls: deciding without past cases, DeepSeek V4.1 Flash approved each in some of four tries and denied it in others. When the past cases are near-copies of the request (its facts with a few details changed), switching their recorded decisions from DENY to APPROVE raises approval by about 30 to 55 percentage points in DeepSeek V4.1 Flash, GPT-5.6 Luna and Claude Haiku 5.5 on policies written for the study, and DeepSeek and Luna show the same on nine public policies paraphrased from real ones. The facts push the other way: shown without their recorded decisions, past cases whose facts support approval make these two models approve less often than past cases whose facts support denial. One reading is that the request looks weaker next to them. This push is not just a correction toward the right answer: in an additional DeepSeek analysis, it was of similar estimated size on requests that human annotators judged approvable and on those they judged deniable. In a separate DeepSeek run on the public policies, when the past cases are near-copies of a different request, switching their decisions moves approval by 9 points instead of 43, while facts that support approval still lower it by 22 points against facts that support denial. Systems that retrieve past decisions should test the two parts separately.