Language adds limited value to vision models in pathology tasks
What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Computational pathology models combine pictures of tissue with written reports to help predict patient outcomes or answer questions. The authors tested how much adding language really helps versus just using images alone. They found that report text offers small improvements in predicting patient survival but doesn’t help with certain key details. Also, using language prompts or questions often doesn’t improve accuracy much, and answers sometimes change if the questions are reworded. Overall, language adds some value but mostly in narrow ways for these medical tasks.
What this means in practice
- •For hospital data teams: Use slide embeddings combined with report embeddings to slightly improve patient survival predictions from pathology data.
- •For medical ai developers: Evaluate vision-language models for cancer subtype classification with careful controls, avoiding overreliance on language prompts during inference.
Authors
Wenhao Zhang, Zhongliang Zhou, Yi Ping, Zirui Wang, Sheng Li
Abstract
Vision--language models (VLMs) promise to make computational pathology queryable and able to draft reports, but how much language adds beyond the vision backbone is unclear, because almost every system is evaluated by its developers on test sets of their own choosing. We grade published evidence by how it was produced and test four claims with controlled probes that reuse released checkpoints and public data, varying one input at a time. Report text adds prognostic signal: adding a frozen embedding of the stage-redacted report to a slide embedding raises the pooled C-index by 0.019 for progression-free interval and by 0.016 for overall survival across 1{,}633 TCGA patients, and restoring the stage statement adds nothing. Prompts add no measurable accuracy at inference: with features and pooling fixed, zero-shot classification is statistically indistinguishable from a supervised linear probe for lung and renal cancer subtyping. Slide-level question answering is barely grounded: SlideChat answers 74.82\% of SlideBench-VQA(TCGA) items correctly with the slide and 74.41\% without it, whereas its language backbone alone reaches 43.52\%. Paraphrasing leaves accuracy unchanged but flips 8--23\% of answers. Self-reported gains from agentic navigation and few-shot language priors cannot yet be attributed to language. Language thus adds real but narrow value, and we propose minimum controls that would expose these failure modes in future evaluations.