Llms improve molecular tumor board safety by reducing over-refusal errors
OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
Artificial IntelligenceMachine Learning
Summary
Molecular tumor boards help doctors decide cancer treatments by combining genetic and clinical information. The authors found that large language models often refuse to recommend treatments even when there is some evidence, leading to unnecessary caution. They created a dataset and a new system, MTB-AuditAgent, that better distinguishes risky from appropriate recommendations. This system reduces over-cautious refusals while keeping safety high, helping AI better support oncologists in treatment decisions.
What this means in practice
- •For oncology clinical teams: Assist clinicians by providing AI-generated treatment recommendations that carefully balance evidence support and incomplete clinical details in complex cancer cases.
- •For medical software developers: Incorporate the MTB-AuditAgent framework within clinical decision support tools to improve safety classification and reduce over-rejection of treatment options by AI models.$Commercial implications: Enables development of safer AI-assisted oncology software by reducing unnecessary refusals and aligning AI recommendations with clinical needs.
Authors
Negin Ashrafi, Jia Luo, Stacey M. Frumm, Roxana Daneshjou
Abstract
Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology. As AI enters this workflow, a key safety challenge is distinguishing truly unsupported recommendations from evidence-supported options that still require oncologist review because of incomplete information, poor ECOG performance status, or other clinical caveats. We introduce OpenMTB-Audit, an open-source benchmark of 500 synthetic non-small cell lung cancer cases spanning five adversarial error categories and four safety labels: Supported, Partially Supported, Unsupported, and Insufficient Information. Across eight large language model configurations, we identify pervasive over-refusal: all LLM configurations failed to retain the Partially Supported label in 83.3-100% of true Partially Supported cases, achieving high aggregate safety scores through label collapse rather than clinically calibrated reasoning. To address this limitation, we developed MTB-AuditAgent, a deterministic seven-module framework separating evidence verification, missing-information detection, safety classification, and abstention. It reduces over-refusal to 6.7% and achieves 91.2% accuracy (95% CI: 88.6-93.6%). A two-oncologist annotation study found disagreement concentrated at the boundary between information sufficiency and treatment optimization, underscoring the need to preserve clinically meaningful distinctions.