Traditional Chinese medicine exam questions benchmarked with expert answers
TCMQA: A 38K-Question Traditional Chinese Medicine Benchmark with a Licensed-Practitioner Reference
Computation and LanguageArtificial Intelligence
Summary
Most medical language tests focus on Western medicine, but Traditional Chinese Medicine (TCM) works differently and has its own tests and knowledge. The authors created a big set of over 38,000 TCM exam questions with answers from experienced TCM doctors. They tested many AI language models on these questions and found none of them mastered the material yet. Some models trained on Chinese data did better than those trained on Western medicine, and some even outperformed the typical doctor’s group answer. However, models and doctors disagree on which questions are hard, and sometimes doctors get questions right that AI does not.
What this means in practice
- •For health technology companies: Build AI tools specialized in Traditional Chinese Medicine by leveraging a comprehensive dataset of TCM exam questions and licensed practitioner answers to enhance diagnostic or educational tools.$Commercial implications: Enables development of accurate TCM-focused AI products for medical education and patient consultation services.
Authors
Tzu-Heng Huang, Jet Lin, Eric Lin
Abstract
Medical benchmarks for language models are built almost entirely on Western biomedicine. Traditional Chinese Medicine (TCM) is a separate system, with its own diagnostic framework and its own literature, and it remains largely unmeasured. The few TCM evaluations that exist are small, narrow, and rarely paired with a human reference. We present TCMQA, an open benchmark of 38,279 questions from Chinese TCM licensing examinations, paired with 15,151 responses from 101 licensed practitioners. We evaluate 29 instruction-tuned models from 9 families, spanning 0.27B to 14.8B parameters. Accuracy ranges over 59 points, and no model approaches saturation. Pretraining data predicts TCM ability far better than scale: a 12B Western-pretrained model reaches 39.6%, while a Chinese-pretrained model an eighth its size reaches 60.8%. Nine models exceed the practitioner majority vote of 64.9%, the best by 21.8 points, and all nine come from that same Chinese-pretrained family. Yet difficulty does not transfer between models and practitioners: accuracy is flat across practitioner-rated difficulty, item-level agreement is near zero for all 29 models, and on $8.4\%$ of items the practitioners are correct where the leading model is wrong. We release the corpus, the practitioner responses, the harness, and per-item model outputs at https://huggingface.co/datasets/TechTCM/TCMQA.