PathCowork trains AI by watching how pathologists work slides
PathCowork: Learning from real-world pathology practice to build AI copilots
Computer Vision and Pattern Recognition
Summary
Medical AI usually learns only from the final answers doctors give, not how they examine slides to reach those answers. The authors created PathCowork, a system that records pathologists’ eye movements, spoken explanations, and navigation through slides to teach AI how to better interpret medical images. They collected detailed data from many pathologists reviewing microscope slides and used it to build AI agents that understand visual cues and reasoning steps. These AI helpers performed better and faster than existing models on several organ-specific tasks, showing promise for learning directly from doctors’ real-world work.
What this means in practice
- •For hospital it teams: Deploy AI assistants that understand pathologists’ workflows to improve diagnostic accuracy and speed in clinical slide review.
- •For medical device developers: Create new pathology AI tools that learn from doctors’ real-time examination behaviors and explanations for enhanced interpretability.$Commercial implications: The paper enables building AI pathology copilot products that incorporate expert visual and procedural data, differentiating them from standard AI diagnostics.
Authors
Sheng Wang, Jialu Yao, Maria V. Rossi, Shunsuke Koga, Esther Baranov, MacLean P. Nasrallah, Songhao Li, Jonathan Xu, Wei Du, Cindy McGrath, Timothy Chao, Anne Wang, Zhaohai Yang, Daniel Akuma, Selemon M. Walle, Olga Pozdnyakova, Edward B. Lee, Zahra Alipour, Annapurna Saksena, Rashmi Tondon, Nikolina Dioufa, David E. Elder, George J. Netto, Xiaowei Xu, Zhi Huang
Abstract
Medical AI learns from the answers doctors record, but rarely from how they interpret the evidence to reach them. Here we introduce PathCowork, a framework that turns pathologists' narrated examinations into supervision for visual grounding and procedures that guide diagnostic AI agents. Synchronized gaze, narration and navigation link visual findings to the process of interpreting them. Across 48 sessions from 16 pathologists, the platform captured 1,028 minutes of slide review. Using these data, we trained PathCowork-1T for server deployment and PathCowork-27B for local deployment, with improved visual grounding. Across five organ-specific evaluations, agents combining this grounding with recorded diagnostic procedures achieved primary task scores averaging 17.9 and 24.1 percentage points higher than Claude Code and Codex, respectively, with shorter processing times and lower inference costs. Compared to the state-of-the-art multiple-instance learning AIs, the PathCowork agent-proposed regions achieved higher prostate-grading scores with limited slide labels. PathCowork can also help construct pathology AI benchmarks where questions derived from the recordings identified difficulties in recognizing viewing context and rejecting inapplicable premises. In summary, PathCowork proposes a new paradigm to scale up the data collection and train pathology AI through transparent reasoning from diagnostic sessions, opening a path towards systems that learn directly from routine clinical practice.