Video diagnosis improves tests and accuracy in neurological cases
DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Diagnosing patients by watching videos involves more than just spotting symptoms; it requires deciding what questions and tests to run next. The authors studied video-based diagnosis in neurological consultations and found that simply recognizing symptoms or watching videos in order doesn't explain accuracy gains. Instead, the key improvement comes from deciding which additional tests to perform based on video observations. They show that helping AI models describe video signs better and retrieve relevant medical knowledge leads to more accurate diagnoses.
What this means in practice
- •For clinical ai developers: Improve AI diagnostic tools by integrating video summarization and targeted medical knowledge retrieval to guide better clinical test selection.
- •For medical software teams: Build video-based diagnostic aids that enhance clinician decision-making through automated sign description and evidence-driven hypothesis testing.
Authors
Jiahui Li, Yutong Guo, Nan Yang, Wenzhan Song, Jin Lu, Fei Dou
Abstract
Diagnosing a patient from video requires more than recognizing the sign: a vision-language model must turn what it sees into hypotheses, questions and tests. DynamicDx evaluates each step in 71 neurological consultations across 11 sign categories, linking authentic patient videos to confirmed diagnoses and fixed charts built from the same case reports, so that every model queries the same evidence. Across five such models, video improves accuracy by 9.9-22.5 percentage points over blind input, but neither recognition alone nor temporal order explains the gain: the cause is usually missing from the model's video-only differential diagnosis even when the sign is recognized, and shuffling the frames produces no reliable accuracy loss. Instead, a trajectory replay traces most of the gain to the investigation results the video prompts. Evidence acquisition is the bottleneck: supplying the decisive investigations raises accuracy to 73.2-93.0%. Two interventions act on it. A post-trained 4B video describer improves sign descriptions, especially from a short, densely sampled segment, and source-clean literature retrieval expands initial hypotheses; both bring the tests a model orders closer to those the treating clinicians documented and, through them, raise accuracy. For video-based diagnosis, seeing better helps when it leads to asking better.