Vision-language models tested on microscopic cell tracking tasks
CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Tracking how cells move and change over time is tricky and usually needs specialized tools. The authors created a big test set called CellTrack-Bench to see how well large vision-language models (which understand both images and text) can do this job compared to specialized software. They collected many cell tracking problems from different types of microscopy images and focused on the hardest cases where current trackers struggle. They then tested various popular models and analyzed where they fail, providing tools and data for others to try as well.
What this means in practice
- •For biomedical imaging teams: Evaluate and compare the accuracy of vision-language models against traditional specialized cell trackers using a comprehensive benchmark.
- •For medical ai developers: Develop improved AI tools for understanding cell movement by using benchmark failures to identify weaknesses of existing models.
Authors
Ilan Theodoro, Jordão Bragantini, Loïc A. Royer
Abstract
Large language models have shown outstanding performance on text-first tasks such as coding, math, and writing. Recent efforts are shifting models' training to include scientific problems that often live in highly specialized domains beyond text, such as protein structure and biological images. However, in these scenarios, their general-purpose accuracy compared with specialized algorithms remains in question. One such domain is cell tracking, which requires spatiotemporal awareness of cell dynamics. To evaluate this, we created a new benchmark, CellTrack-Bench, which assesses Vision-Language Models' (VLMs) cell-tracking-related performance against specialized trackers. The full benchmark draws from 15 standard cell-tracking datasets across four imaging modalities and comprises 1,537 problems corresponding to 9,041 cell-linking decisions. We selected a hard subset requiring greater reasoning effort by identifying failures shared across distinct algorithmic trackers. With this proposed dataset, we: (i) evaluated different environment harnesses for VLMs; (ii) benchmarked 9 frontier closed-source models and 13 open-weight models; and (iii) provided a detailed analysis of their failure modes. Code, the stored model outputs, and a pipeline that regenerates the benchmark from the source datasets are available at https://github.com/royerlab/celltrack-bench.