Medical reasoning improves with targeted training and rubric rewards
Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Medical AI models often struggle to correctly diagnose illnesses and handle complex doctor-patient conversations. The authors designed a new training method that teaches models first how to diagnose using special questions, then how to handle multi-turn medical dialogues using detailed scoring guidelines called rubrics. This approach made the AI better at both kinds of tasks, outperforming some top existing medical AI models. The improvement shows how carefully crafted training can teach AI to reason more like human doctors.
What this means in practice
- •For healthcare ai developers: Build more accurate AI systems that can diagnose diseases and conduct detailed patient conversations using targeted synthetic training and rubric feedback.$Commercial implications: Enables development of clinically reliable AI assistants that outperform existing products in diagnostic accuracy and patient interaction.
- •For medical chatbot teams: Improve clinical dialogue systems by training on synthetic multi-turn scenarios with rubric-based evaluation to handle complex patient interactions.
Authors
Kalash Shah, Kunal Singh, Snehan J, Shreyas Singh
Abstract
Deploying Large Language Models (LLMs) in healthcare requires robust performance across two complementary dimensions - diagnostic reasoning: the convergent, evidence-driven task of inferring a patient's condition from clinical data to produce a diagnosis, and clinical healthcare reasoning: the broader, navigational judgment required to communicate, plan, and adapt across multi-turn clinical interactions where a single correct answer may not exist. Recent benchmarks such as HealthBench and MedXpertQA reveal persistent weaknesses in both areas, exposing failures in complex diagnostic scenarios and limitations in contextual, patient-centered dialogue. We introduce a sequential training framework that targets these facets using synthetic data and rubric-based reinforcement learning. First, we improve diagnostic reasoning using MedBullets-derived questions with rule- and rubric-guided Reinforcement Learning (RL). We then shift to clinical reasoning by generating 5.3k synthetic multi-turn scenarios, each paired with multi-dimensional rubrics to comprehensively assess the response. This approach yields over 10% improvement on MedXpertQA, and our 30B model achieves 50.1% accuracy on HealthBench-Hard, surpassing proprietary baselines including GPT-5 (thinking). Our results show that targeted synthetic datasets and rubric-based training can systematically improve both diagnostic and interactive clinical reasoning in medical LLMs.