Embedded code helps language models perform exact medical calculations
Improving Medical Calculation of LLMs with Embedded Coding
Artificial IntelligenceComputation and Language
Summary
Medical language models can answer many questions well but often make mistakes when exact math is required, like calculating medication doses. The authors developed a method called MedCode that teaches these models to write small programs to do precise math calculations. This code runs inside a safe interpreter, ensuring the answers are exact and come with explanations and correct units. Tests show this method improves accuracy by 20 to 30 percent on important medical calculation tasks.
What this means in practice
- •For clinical software developers: Integrate embedded executable code generation into medical language models to improve accuracy of dosage and prognostic calculations in healthcare apps.
- •For icu data teams: Use the embedded code framework to enhance automated calculation reliability in intensive care unit monitoring and decision support systems.
Authors
Tianshi Ming, Yingying Zhang, Xian Wu
Abstract
Large Language Models (LLMs) perform well on medical examinations and question-answering benchmarks, but remain unreliable on medical calculation tasks that require exact numerical outputs. These calculations support high-stakes decisions such as medication dosing, organ-function assessment, and prognostic scoring, for which even small errors can have serious clinical consequences. We introduce MedCode, a framework that improves medical calculation by training LLMs to generate embedded executable code. Given a clinical context, the model identifies the relevant calculator, extracts its input variables, and produces a script that delegates arithmetic operations to a deterministic interpreter. Executing the script returns the calculated value together with an explanation and the appropriate unit. We construct supervised fine-tuning (SFT) and preference datasets from the MedCalc benchmark and additionally curate a dataset for calculation tasks in Intensive Care Unit (ICU) scenarios. We further propose weighted Direct Preference Optimization (wDPO), which adaptively emphasizes preference pairs that are difficult for the model to distinguish. Experiments with LLaMA3-8B, Qwen2.5-7B, and Mistral-7B show absolute accuracy gains of 20--30 percentage points, demonstrating the effectiveness of embedded code generation for medical calculation.