Open-weight large language models generate points for electron microscopy segmentation
Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentation
Computer Vision and Pattern Recognition
Summary
Electron microscopy images need to be segmented to study tiny structures, but often this requires humans to click points on the image first. The authors show that certain large multimodal language models can generate these point prompts automatically simply by reading natural language instructions. They train the models on datasets of mitochondria images with known points and improve their accuracy for segmentation. This approach means the language model can link descriptions directly to image regions, helping automate segmentation without human input.
What this means in practice
- •For electron microscopy imaging teams: Enable fully automatic segmentation of mitochondria by generating point prompts from natural language, reducing manual annotation effort.
- •For biomedical image analysis developers: Incorporate open-weight multimodal language models to produce localization points for segmentation pipelines on new or unknown EM datasets.
Authors
Samia Mohinta, Albert Cardona
Abstract
Promptable models such as microSAM segment electron microscopy (EM) images from point prompts, but automation requires generating prompts without user input. We ask whether open-weight multimodal large language models (MLLMs) can generate them from natural-language requests by returning coordinates to a frozen segmenter. To that end, we convert masks from three mitochondria datasets into training examples, pairing images and instructions with centroid coordinates, then train LoRA adapters while freezing the MLLM backbone and microSAM. We find that Qwen3-VL reaches segmentation AP$_{50}$ $0.736$ after supervised fine-tuning and reward optimization, up from $0.247$ without adaptation, while automatic prompt generation (APG) achieves $0.773$. In addition, two other MLLMs improve, reaching or exceeding APG. When compared with a supervised centroid-heatmap detector that reaches AP$_{50}$ $0.904$ for this mitochondria task, Qwen3-VL more closely matches the annotated point set and instance counts. Moreover, training on two public datasets transfers to an unseen third, while training on all three transfers to an independent EM volume. Robustness tests show stable performance under unseen formulations of the natural-language request, while the coordinates can be reused by a second segmenter. To our knowledge, this is the first feasibility study of open-weight MLLMs as EM point generators, providing an inspectable, language-directed link between localization and mask decoding.