Jev model improves clinical triage probability accuracy and safety
Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
Machine Learning
Summary
Accurate advice in patient triage is important, especially when more information might be needed. The authors tested Jev, a model that gives probabilities for different advice options, to see if it helps make better decisions and know when to ask questions. Jev performed better than other tested models in giving well-calibrated probabilities and reducing mistakes when deciding if enough info was available. However, asking separately about information needs helped catch more unclear cases but caused too many unnecessary questions.
What this means in practice
- •For clinical decision support teams: Incorporate Jev model outputs to improve accuracy and confidence in automated patient triage recommendations and detect when more patient information is needed.
- •For healthcare chatbot developers: Use Jev’s calibrated probability outputs to enhance triage chatbots’ ability to give reliable care advice and recognize cases requiring clarification.
Authors
Liang You, Zhiyin Zhou, Dawei Tang, Chuchu Wu, Jingzhuo Hu
Abstract
Patient-facing triage requires an appropriate care recommendation and recognition of when information is insufficient. We evaluate whether Jev, a decision model that returns probabilities over predefined actions, supports both judgments. Using CARE-Bench, we compare Jev with four general-purpose language models and three input-independent baselines across 925 decision points from 439 public cases. On the public test subset, Jev ranks fourth of five models by macro-F1 (0.461) but has the lowest observed multiclass Brier score (0.610), an 18.7\% improvement over uniform prediction (case-bootstrap 95\% interval, 4.9--31.1\%). Noul has lower binary Brier than Choice (0.215 versus 0.276). A evaluation of 453 HealthBench conversations again gives Jev the lowest observed Brier score (0.299), but measures a distinct emergency-referral judgment. Jev combines lower probability error with missed clarification needs in the four-action task. Asking about information sufficiency separately increases recall but also produces frequent unnecessary clarification assignments.