Co-Annotator: Expert-Distilled ViT and VLM for Visual and Documentation Guidance in Age-Related Macular Degeneration
2026-08-31 • Artificial Intelligence
Artificial IntelligenceHuman-Computer Interaction
AI summaryⓘ
The authors created Co-Annotator, a tool that helps doctors analyze eye scans by showing them important areas to focus on and suggesting notes automatically. They first trained the system using eye movement and speech data from experts, which improved its accuracy. When tested with eye doctors in training, the tool made both finding problems and writing reports easier and faster, especially when both features were used together. Overall, their approach reduces the time doctors spend searching images and writing, without lowering diagnostic quality.
Vision TransformerRetinal Optical Coherence TomographyExpert Gaze TrackingVision-Language ModelBiomarker DocumentationVisual Search EfficiencyClinical WorkflowOphthalmologyMultimodal GuidanceDiagnostic Accuracy
Authors
Ziheng "Leo" Li, Benjamin Freeman, Akshay Raman, Kavin Aravindhan Rajkumar, Xinxin Fang, Rishabh Srivastava, Steven Feiner, Kaveri A. Thakoor
Abstract
Clinical AI often optimizes predictive performance without engaging how clinicians decide where to look and what to write. We present Co-Annotator, which distills expert gaze and dictation into two guidance components: a gaze-aligned Vision Transformer producing fixation-aligned areas of interest (AOIs), and an ontology-bounded vision-language model (VLM) that pre-fills editable biomarker summaries for retinal optical coherence tomography (OCT). We first collect expert gaze and dictations (US1) to train the models, significantly improving diagnostic accuracy and biomarker generation. We then deploy the system with ophthalmology residents: a controlled resident study (US2) confirmed each modality is safe and independently beneficial, with AOI guidance producing lasting perceptual efficiency gains through post-guidance carryover and VLM guidance more than doubling biomarker documentation breadth. In a combined deployment across two academic institutions (US3), providing both modalities simultaneously produced efficiency gains that substantially exceeded either modality alone: correct diagnoses per minute increased by 40% and comment editing time fell by 67%, without compromising diagnostic accuracy. Notably, neither modality improved efficiency during guidance in US2, which makes the in-guidance efficiency gain under combined guidance in US3 the more striking result. Expert-distilled multimodal guidance can remove two distinct clinical workflow bottlenecks at once (visual search overhead and documentation burden) without compromising the diagnostic accuracy clinicians already achieve.