Language models explanations often fail to guide denied request fixes

Right Reason, Wrong Advice: Evaluating Recourse in LLM Explanations of Denied Requests

Artificial IntelligenceComputation and LanguageComputers and Society

Summary

When a computer denies a request based on many rules, it can explain the reason, but fixing that reason might not help get approval. The authors show that explanations often lack "recourse validity," meaning a person can't change the outcome by addressing the cited reason alone. They tested many models and found that most fail to detect when explanations don’t provide a helpful path to success. Better explanations come from linking explanations closely to the decision rules and checking if acting on them would actually help.

What this means in practice

Authors

Arunkumar V

Abstract

Language models increasingly write the explanations people receive when a rule system denies their request. If several rules deny the request independently, an explanation can name a true reason and still leave the person denied after they fix it. We call the missing property recourse validity: changing what the explanation cites, without changing what was asked for, can flip the decision. We compute it exactly, and a hitting-set argument predicts from the policy alone when a correct reason fails. In six of seven rule systems ported from access control and US benefit eligibility, a single correct reason almost never gives recourse when several reasons apply. Across 20 LLMs, checks of whether the cited reasons are correct, computed or asked of an LLM judge, are close to chance at spotting this failure, as the argument predicts. A strong judge asked whether acting on the explanation would work does spot it. Stronger and reasoning models do better but still fail on many denials with four or more independent reasons. Asking for every rule helps every open model with room to improve. Explanations of eligibility decisions should be built from the decision function and judged by whether acting on them works.