Vision guided retrieval improves medical time series analysis accuracy
Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective
Computer Vision and Pattern Recognition
Summary
Medical data often comes as numbers changing over time, like heartbeats or brain waves. Usually, these numbers are analyzed without looking closely at their shapes, which doctors often use when reading waveforms. The authors created a method called Vision-Informed Retrieval (ViRe) that uses pictures of these waveforms and combines them with the numbers to better understand the data. Their method picks out the most important parts of the numerical data by looking at the waveform shapes through a vision-language system. This approach performed better than previous methods on multiple medical datasets.
What this means in practice
- •For hospital data teams: Improve clinical diagnosis systems by integrating waveform shape information with numerical patient time series data for more accurate classification.
- •For medical device developers: Develop diagnostic tools that automatically combine waveform images and raw signal data to enhance detection of medical conditions.$Commercial implications: Enables new products that provide enhanced signal interpretation using vision-language integration for medical diagnostics.
Authors
Guoqi Yu, Juncheng Wang, Shujun Wang
Abstract
Medical time series (MedTS) underpin many clinical classification tasks, yet existing methods usually represent them only as numerical sequences and underuse the morphology that is explicit in waveform inspection. To bridge this gap, we introduce Vision-Informed Retrieval (ViRe), which uses a frozen VLM-derived waveform representation as a morphology-aware Query to guide retrieval from raw numerical MedTS features. Specifically, a Vision Query is extracted using pre-trained vision-language models (VLMs) to obtain morphology-aware priors from waveform plots. A tailored attention-based cross-modal retrieval mechanism then uses the Vision Query to select morphology-relevant temporal and channel evidence from the numerical representation. ViRe demonstrates strong effectiveness against ten established baselines, yielding an overall 6.42% relative improvement over the previous state of the art across six public benchmarks. Code, training scripts, and reproducibility materials are publicly available in the GitHub Repository: https://github.com/Levi-Ackman/ViRe.