Proxy reliance in large language model decisions is uncalibrated to predictive evidence
2026-08-24 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageComputers and Society
AI summaryⓘ
The authors studied how large language models (LLMs) make decisions in sensitive areas like healthcare, focusing on whether these models wrongly use unfair proxy information related to protected groups. They developed a way to measure if the models rely too much, just enough, or too little on such proxies using a clinical task where the correct reasoning is known. Their results showed that all models sometimes use meaningless proxy information, and efforts to reduce this reliance aren't very strong or consistent. They also found that standard accuracy checks don't reveal these issues.
large language modelsproxy usefairness auditingcausal inferenceclinical decision-makingprotected groupsalgorithmic biasmodel evaluationin-context learning
Authors
Zengqing Wu, Chuan Xiao
Abstract
Large language models (LLMs) are entering decisions in triage and lending, where task-relevant inference must be distinguished from impermissible proxy use. Current audits ask whether decisions change when demographics change. But attributes correlated with a protected group carry predictive value, so a changed decision can be discrimination or sound inference. We measure causal proxy effects in four LLMs on a clinical-ranking task with known ground truth, where the reliance the evidence warrants can be computed exactly and used as the reference. One audit signal yields three verdicts: over-reliance, warranted and under-reliance. Under neutral labels every model relies on proxies with no information. Informative proxies draw all three. Social field names push reliance down, below the reference in one model. Two findings explain this. Reliance severely undertracks the evidence, and social-label suppression is fragile, since in-context examples raise it above zero in every model. Accuracy-based evaluation detects none of this.