Language models show limits reasoning prescriptions from Chinese medicine evidence
TCMClinicalReason-Bench: Can Language Models Reason from Pathogenesis to Prescription over Real-World Clinical Cases?
Artificial Intelligence
Summary
Large language models can write clinical notes but often make mistakes when basing decisions on patient details. The authors created a benchmark using real-world traditional Chinese medicine cases to test if these models correctly reason from causes of illness to prescriptions. They found models are good at structuring responses but less accurate at supporting their conclusions with evidence, especially in generating prescriptions. This shows where these AI tools need human oversight to avoid errors in medical treatment decisions.
What this means in practice
- •For clinical ai developers: Evaluate and improve AI tools to generate evidence-supported diagnoses and prescriptions in traditional Chinese medicine cases.
- •For clinical quality assurance teams: Identify potential failure points in AI-generated clinical reports to guide necessary clinician review and oversight.
Authors
Jirui Dai, Chenkai Zhang, Yan Jia, Yukai Wang, Ruiyang He, Changyong Luo, Zhi Liu
Abstract
Large language models (LLMs) can generate clinical narratives that are insufficiently grounded in patient-specific evidence. In traditional Chinese medicine (TCM), errors can propagate from etiology and pathogenesis through syndrome diagnosis and treatment principles to prescription generation. We developed TCMClinicalReason-Bench using 2,000 multicenter electronic health record cases to distinguish case-grounded responses from fluent but unsupported diagnostic and therapeutic conclusions. Five general-purpose and two TCM-specific LLMs were evaluated in zero-shot settings. An evidence-constrained rubric assessed seven diagnostic and therapeutic components and three cross-block relations, allowing case-supported alternatives. Qwen3.7-Plus with TCM retrieval served as the automated judge, alongside parallel blinded ratings by five senior TCM clinicians on a 600-case subset. Structural completeness was nearly saturated (99.3-100.0%), but normalized content scores ranged from 40.7% to 54.1%. The five general-purpose models averaged 50.0%, versus 41.3% for the two smaller TCM-specific models. Cross-block logic consistency ranged from 60.8% to 66.8% and correlated moderately with content across cases (Pearson's r = 0.515-0.656). Deficits were greatest in prescription generation, prescription analysis, and symptom-guided modification. In judge stress testing on 100 independent cases, perturbation detection rates across the three relations were 57%, 56%, and 31%, with contradictions detected more reliably than omissions. Separating component quality from cross-block consistency localizes failures missed by endpoint and completeness metrics and identifies where clinician oversight remains necessary.