Large language model audits clinical prediction models with high accuracy

Auditing black-box clinical prediction models with a large language model through queries alone: a blinded test with planted flaws

Machine LearningArtificial Intelligence

Summary

It is hard to check if computer models used in medicine are working correctly or if they have hidden problems. The authors trained different medical prediction models and secretly added mistakes to some of them. They then asked a large language model, GPT-6 Astra, to find these mistakes just by looking at the models' predictions. The model found almost all the hidden flaws and gave detailed explanations, although some of its extra findings needed further checking. This shows that large language models can help safely review complex medical prediction tools without needing full access to their inner workings.

What this means in practice

  • For hospital data teams: Check clinical prediction models for hidden errors using only queries without inspecting their code or training data.
  • For ai monitoring teams: Use large language models to audit complex AI systems' output behavior for faults and inconsistencies when internal details are unavailable.

Tested on one dataset.

Authors

Zhiyu Liu, Xinyu Yang

Abstract

Large language models (LLMs) are used to explain clinical prediction models, on the promise that they can say what a model computes, find its faults and even reveal new knowledge in it. We tested the three promises where the answers could be checked. From the International Stroke Trial we trained four kinds of model (logistic regression, explainable boosting machine, gradient-boosted trees, neural network) for two outcomes on patients randomised earlier, planted one flaw in each of 24 copies (an under-predicted subgroup, a reversed risk factor or an implausible step) and mixed them with the 8 clean models. GPT-6 Astra audited each model three times, blind, through a tool that only returned the model's predictions; the judging rules were committed before any run and applied by program. Its surrogate models, with a median of 128 terms, reproduced the audited models' log-odds with a mean $R^2$ of 0.976 (0.959 for an additive surrogate of 42 terms fitted by an analyst). It found the planted flaw in 69 of 72 runs (96%; 95% CI 80% to 99%); a script written for the purpose found all 24. It also reported many more problems: 60% of 1,089 checkable reports failed the pre-specified check against the model's average behaviour; post hoc, most of these were real but small steps, and 15% of the checkable reports were not found in the model. Of 601 unexpected relationships it read out of the clean models, 11% held in patients randomised later (random pairs 6%; a regression scan's shorter list of its strongest candidates, 28%); after a check on the training data, 33% against 30%. With query access alone, an LLM can describe a black-box model faithfully and find nearly every planted flaw, as a purpose-written script also did; its reports of problems must be checked on the model and its discoveries on data.