Tool augmented language models improve reliable medical calculations from notes
Tool-Augmented Large Language Models for Reliable Medical Calculation from Patient Notes
Artificial Intelligence
Summary
Medical calculations often require precise numbers and specific steps that are hard to get from messy doctor’s notes. The authors found that relying on language models alone to do these calculations can lead to mistakes. They built a system that breaks the job into parts: it first pulls out the needed information clearly, then uses special tools to do the math exactly. This helps avoid errors and makes medical scoring more reliable in patient records.
What this means in practice
- •For hospital data teams: Automatically convert free-text patient notes into accurate clinical scores for better decision support.
- •For healthcare software developers: Create reliable medical software that uses executable calculation tools to reduce errors from language model outputs.$Commercial implications: Enables development of commercial clinical decision tools with higher accuracy by combining language understanding and precise computation.
Authors
Yang Zhong, Ling Luo, Xuanlyu Zhu, Yanhua Wang, Yuanyuan Sun, Jian Wang, Hongfei Lin
Abstract
Medical calculators translate raw patient observations into clinically actionable values and scores. However, applying these calculators directly to free-form patient notes requires precise variable extraction combined with the exact execution of complex formulas, scoring rules, and output conventions. Relying solely on direct large language model (LLM) prompting is unreliable in this context, as rule recall, parameter extraction, arithmetic reasoning, and formatting become entangled within a single generative step. To address these limitations, we propose a tool-augmented LLM framework for reliable medical calculation from patient notes. The framework retrieves and structures calculator specifications, generates reusable executable tools, and uses a constrained extractor to map each patient note to calculator-ready inputs. The tools then perform deterministic calculation, separating clinical language understanding from rule execution and arithmetic. Evaluated on the MedCalc-Bench dataset, our method achieves 86.73% final-answer accuracy, outperforming the one-shot prompting and supervised fine tuning-only baselines by 24.0 and 17.27 percentage points. Ablation and error analyses show that these performance gains are primarily driven by rule-grounded tool synthesis and schema-constrained extraction, whereas remaining failures are concentrated in clinical-semantic extraction and benchmark-specific output contracts. By grounding calculator logic in executable tools, the proposed approach alleviates arithmetic hallucinations and improves the computational reliability of the final clinical outputs.