Australian clinical AI models benchmarked for current policy accuracy
AusClinBench: a deterministically scored benchmark of large language models against current Australian clinical policy
Computers and SocietyComputation and Language
Summary
AI models are increasingly used in Australian healthcare without clear information on which model is being used. The authors created 142 questions based on Australian clinical policies to test 28 AI models for accuracy. They found accuracy varied widely and often was below perfect, especially when models had to recall or apply local rules. The benchmark helps assess how well models follow Australian healthcare guidelines and suggests that models should be evaluated as part of clinical systems before use.
What this means in practice
- •For clinical software developers: Evaluate base AI models for compliance with Australian clinical rules before integrating into healthcare applications.
- •For health it product managers: Set minimum standards on model accuracy for local policy adherence alongside broader product quality assessments.
- •For insurance claim processors: Use the benchmark to improve automated billing review accuracy by verifying models on correct application of Australian billing rules.$Commercial implications: Enables development of automated billing tools tailored to Australian rules that reduce errors and claim rejections.
Authors
Daniel S. Epstein
Abstract
AI models already inform clinical decisions and draft notes in Australian clinical software, usually without the practitioner being told which model is used. Existing medical benchmarks are written for other jurisdictions. We wrote 142 questions from Australian guidelines and schedules across five domains (PBS authority, MBS billing, Ahpra obligations, triage red flags, immunisation and screening), each with one verifiable answer scored by exact match. Each question ran blind and with the governing clause supplied, which separates failure to recall the current rule from failure to apply it. We tested 28 models between July and October 2026. Blind accuracy ranged from 51.5% to 92.7%, and the highest-scoring models could not be separated. Of 791 failed model-question pairs, 684 (86%) were wrong blind and right with the clause. Confident-error rates ranged from 6.6% to 47.8%. On five immunisation questions where the guideline gives no directive, 82% of answers stated one. On 20 Aboriginal and Torres Strait Islander health questions, models did no worse overall, but on a few the wrong answers gave the rule for the general population, an older rule or another country's rule. We propose treating the model as an ingredient of a clinical product, with a minimum standard on local rules that sits beside product evaluation. The keys are one general practitioner's and are not yet independently reviewed. The benchmark measures base models, not clinical products, and does not license any model for clinical use.