VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction
2026-08-31 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors address a problem in clinical AI fairness audits where the pass-or-fail decisions can change over time or between hospitals, causing instability. They note that current methods handle uncertainty in continuous fairness scores but do not directly measure how stable the final verdicts are. To fix this, the authors introduce VFR-Audit, a new method that quantifies how likely a verdict is to flip using a metric called Verdict Flip Rate (VFR). Their approach also evaluates reliability through resampling stability, audit size sensitivity, and agreement between hospitals.
clinical AIfairness auditverdict stabilityVerdict Flip Rate (VFR)bootstrap resamplingbias mitigationAUROCAUPRCFleiss' kappamodel fairness metrics
Authors
Md Jannatul Rakib Joy, Viet Vo, Caslon Chua
Abstract
Fairness audits in clinical Artificial Intelligence convert continuous fairness metrics into binary pass-or-fail verdicts against operational thresholds, where hospital governance boards, payers, and regulators act on the resulting verdicts. Such audits are repeated over time and across hospital sites, thus the same verdict can flip between pass and fail across audits. Existing uncertainty methods such as Bayesian posteriors, bootstrap confidence intervals, and permutation tests address verdict instability only at the continuous-metric level. Converting metric-level uncertainty into a verdict-stability claim remains a manual step that scales poorly across the (model, metric, attribute) cells an audit covers. Existing uncertainty methods also leave open whether bias-mitigation steps, such as reweighing or per-group threshold shifts, yield a stable passing verdict at the cost of model discrimination measured as AUROC or AUPRC.To address this verdict-stability gap, we propose VFR-Audit, a framework built around the Verdict Flip Rate (VFR), a scalar bounded between 0 and 0.5 that measures the probability of verdict reversal under stratified bootstrap resampling. VFR-Audit reports VFR alongside three reliability axes, namely within-cohort resampling stability, audit-size sensitivity, and cross-hospital verdict agreement via Fleiss' kappa.