Jev system shows mixed accuracy on medical question benchmarks
Jev in Medicine: A Benchmark Evaluation. Preliminary Results
Artificial IntelligenceMachine Learning
Summary
Medical question answering needs systems that pick correct answers and know when they don’t know enough. The authors tested Jev, a simple system that only chooses from given options, on four medical question sets and compared it to a smart AI called GPT-6 Sol. Jev was almost as accurate on simpler research abstract questions but less accurate on harder exam and clinical case questions. It was fast and cheap but needs careful testing before being trusted in real medical settings.
What this means in practice
- •For medical ai developers: Use Jev as a fast, low-cost baseline for medical question answering tasks with predefined answer options before developing more complex models.
- •For hospital data teams: Integrate Jev for pre-screening medical research abstracts with reliable confidence scores, optimizing resource use in evidence summarization.
Authors
Alfredo Madrid-García, Beatriz Merino-Barbancho
Abstract
Jev is a non-generative "System One" model that assigns probabilities to predefined answer options and cannot answer outside them. Its accuracy and calibration on medical question-answering and case-based diagnostic-reasoning tasks are unknown. We evaluated Jev 1.13 on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena-MCQ and the NEJM Case Challenges. GPT-6 Sol, with (medium) and without reasoning, was the reference. The primary outcome was top-1 accuracy; key secondary outcomes were calibration, selective prediction and recognition of unanswerable questions. All 8,469 requests returned a valid answer. Jev's accuracy was similar to that of GPT-6 Sol with medium reasoning on PubMedQA (78.4% vs 78.2%;), lower on MetaMedQA (74.8% vs 82.7%) and much lower on DiagnosisArena-MCQ (59.8% vs 82.4%;) and the NEJM cases (61.8% vs 82.4%). On MetaMedQA, Jev's probabilities were the best calibrated (expected calibration error 0.063 vs 0.146), and its answers with a probability of at least 0.9 (52.9% of questions) were 93.4% accurate, but GPT-6 Sol was as accurate when it accepted a similar proportion of questions. On DiagnosisArena-MCQ, Jev's probabilities discriminated poorly (AUROC 0.645 vs 0.768). Of the 162 questions whose correct answer was "I don't know or cannot answer", Jev chose that option for 10.5% (GPT-6 Sol, 8.6%). Median latency was 0.27-0.31 s; all 2,823 items cost USD 0.08. Jev was fast and inexpensive, and its accuracy was similar to that of a frontier LLM on research abstracts but lower on examination questions and much lower on complex diagnostic cases. Task-specific validation is required before clinical use.