Vision language models improve error detection in brain image analysis
Benchmarking Vision-Language Models on Synapse Detection and Proofreading in Connectomics
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Finding connections between neurons in brain images is very hard for people and computers. The authors tested special AI models that can see and understand images along with text instructions to spot connections and fix mistakes. They found that most models don’t do well without help, but a few get better when given some examples or training. The models can also work across different animal brains, sometimes even better than specialized tools. This work helps future AI tools assist scientists studying brain wiring.
What this means in practice
- •For neuroscience data teams: Improve automated detection and correction of neuron connection errors in brain imaging across species using vision-language models.
- •For medical imaging developers: Incorporate vision-language models that adapt with limited labels to better identify structural errors in high-resolution biomedical images.
Authors
Yicong Li, Junjie Wang, Leander Lauenburg, Ella Hugie, Alexandra Irger, Wanhua Li, Donglai Wei, Hanspeter Pfister
Abstract
We benchmarked vision-language models (VLMs) on the decisions annotators take when inspecting electron microscopy images in connectomics: synapse detection (presence and polarity) and proofreading (split errors and merge errors). For synapse detection, we evaluated 19 open and 2 closed models across various architectures and sizes under zero-shot, four-shot in-context learning and LoRA settings, against specialist models, on datasets constructed by us using public resources. For proofreading, we evaluated 3 open and 2 closed models on the ConnectomeBench2 dataset, with cross-species transfer from fly and mouse to human and zebrafish. Most models were at chance zero-shot; a few examples helped mainly the closed and largest open ones. LoRA on a few thousand labels brought open models level with specialist models. When evaluated on unseen species, the best adapted VLMs outperformed specialist models trained on the same data in identifying merge errors. The project will be publicly available upon acceptance.