Traditional Chinese medicine syndrome diagnosis benchmark reveals diagnostic challenges
TCM-DiagCommit: A TCM-SD-Derived Benchmark for Evidence-Conditioned Syndrome Selection and Commitment
Artificial Intelligence
Summary
Diagnosing syndromes in Traditional Chinese Medicine (TCM) usually happens after seeing all patient data at once, hiding important diagnostic steps. The authors created a new benchmark called TCM-DiagCommit that tests how well models pick syndromes as new evidence appears, avoid mistakes when uncertain, and handle confusing options. They show that just checking final diagnosis accuracy misses problems like wrongly committed answers or ignoring new evidence. Their work breaks down diagnosis into smaller tasks to better understand and improve AI in TCM diagnosis.
What this means in practice
- •For healthcare ai developers: Evaluate and improve AI models for stepwise syndrome diagnosis in TCM using staged evidence and commitment scoring.
- •For clinical decision support teams: Integrate more nuanced TCM diagnosis benchmarks to assess system behavior in uncertainty and evidence updates.
Authors
Erwei Hou, Yue Hu
Abstract
Traditional Chinese Medicine (TCM) syndrome differentiation is usually evaluated as static final-label selection over complete clinical records. This setting hides three diagnostic behaviors: whether a model distinguishes semantic-neighbor syndrome labels from easy random distractors, whether it updates predictions as evidence fields are revealed, and whether it should commit under uncertainty. We introduce TCM-DiagCommit, a benchmark derived from the public TCM-SD/ZY-BERT resource while retaining its source syndrome labels; it converts 1,834 source cases into 23,842 task rows over 148 syndrome labels. The benchmark combines random, profile-neighbor, and full-vocabulary candidate ladders with staged t1/t2/t3 evidence. It also includes fixed-label tongue/pulse perturbations and insufficient-evidence variants. A benchmark-defined action contract scores commit, acquire, defer, and abstain. Across five zero-shot DirectLLM calibrations, random10 top-1 accuracy is 63.2% to 66.4%, but hard20 falls to 24.7% to 30.3% and full148 to 14.8% to 22.3%; on hard20, WT(all) wrong-terminal commits remain 62.2% to 74.3%. These results show that final-answer accuracy alone is insufficient for diagnostic benchmark evaluation: answer selection, evidence use, and benchmark commitment scoring should be measured separately.