Benchmark tests language models on medical physics calculations and errors
MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection
Artificial IntelligenceComputation and Language
Summary
Medical physicists carefully check calculations that control radiation treatment to avoid mistakes that could harm patients. The authors created MedPhysBench, a set of realistic medical physics problems and error checks to see how well language models handle these critical calculations. They tested 12 language models on tasks like radiation dose calculations and error detection, finding mixed results with some models nearly perfect and others missing key corrections. The benchmark helps reveal strengths and weaknesses of AI in this precise, safety-critical area.
What this means in practice
- •For medical physics teams: Evaluate language models' ability to assist in verifying complex medical physics calculations and reduce human errors in radiation treatment planning.
- •For quality assurance engineers: Use MedPhysBench to test AI tools designed for automated error detection in clinical radiation dose calculations before deployment.
- •For healthcare ai developers: Develop AI-driven medical physics calculation assistants that improve safety by detecting common errors highlighted in this benchmark.$Commercial implications: Enables creation of validated AI assistance tools for medical physics departments that can be marketed to hospitals for safer radiation therapy.
Authors
Omar Nusrat
Abstract
Medical physicists turn a prescription into a delivered dose and check one another's calculations, because one error in a beam calibration or a monitor-unit (MU) calculation can affect every patient treated with it. MedPhysBench has ten seeded generators of calculation items in radiation therapy, diagnostic imaging, nuclear medicine and radiation protection, up to clinical tasks such as a TG-51 calibration worksheet. Reference answers come from code implementing the governing protocols, and a grader converts units and applies tolerance bands. From the same inputs, separate checking items test error detection, showing a colleague's worksheet or only the colleague's result, half correct and half with one planted error. On a held-out test split of 1,200 items (half for two models), 12 models scored 33.3 to 100.0% on calculation. Of the nine answers graded incorrect for the four best (99.3-100.0%), five contained model errors and four followed NCRP Report 151 on points our items misstate or leave open. Below them, most models did worst on level-3 clinical tasks, and Claude Sonnet 5.5 omitted the inverse-square correction in 19 of 20 isocentric MU items. Five models detected at least 96.7% of mostly large planted errors with at most 4% false alarms; for four of the other seven, checking only the result raised false alarms significantly. In an exploratory targeted test with 20 public cases of an omitted inverse-square correction, the three best models tested caught it in every worksheet, and three weaker models accepted 16 to 18 of 20 planted worksheets, one of them while applying the correction in most of its own calculations. Claude Opus 5.5 accepted 6 of 20 wrong results, although it applied the correction in all 20 of its own. We report the effect of two problems in our shielding items, found after the runs, and release the code, per-item outcomes and targeted-test responses.