Large language models generate medical image classifier parameters without training data

Weight-Space Programming for Medical Imaging: Do We Still Need Training Data?

Computer Vision and Pattern Recognition

Summary

Medical image classifiers usually need lots of labeled images to learn how to recognize diseases, but getting these images is hard. The authors tested whether large language models (LLMs) can create these classifiers just by writing down the network's settings, without seeing any images or training data. They found that the LLMs could build classifiers that perform better than a simpler method but not as well as ones trained with even a small amount of real data. This shows LLMs can help build medical image classifiers but cannot replace the need for training images yet.

What this means in practice

  • For medical ai engineers: Generate initial medical image classifier parameters without any training images to jumpstart model development with scarce labeled data.
  • For image analysis teams: Use LLM-written network weights as a quick baseline for medical image classification tasks across multiple modalities when labels are limited.

Authors

Maximilian Lindholz, Richard Ruppel, Annette Milnik, Sebastian Halskov, Semil Eminovic, Charlie A. Hamm, Robin Schmidt, Anika Knupfer, Jana Hutter, Johanna P. Müller, Carolina Dominguez Aleixo, Jawed Nawabi, Jens Vogel-Claussen, Tobias Penzkofer

Abstract

Can a large language model (LLM) write the parameters of a medical image classifier from medical knowledge alone, without any training data? Such classifiers normally require labelled images, which are costly and often unavailable. We investigate this question through weight-space programming (WSP), in which an LLM writes or adapts the weights and biases of a neural network directly, without optimisation on training data. Claude Opus 5.5 and GPT-6 Astra each generate the parameters of a small LeNet-style CNN in a single reply, without tool use and without access to any image. We compare these networks against zero-shot MedSigLIP and against the same architecture trained on labels using the Adam optimiser, across 7 classification tasks in four modalities: chest X-ray, breast ultrasound and abdominal CT (all from MedMNIST), and pelvic MRI (SynthUterus). The LLM-written networks perform above chance on all six MedMNIST tasks, with a mean test AUC of 0.719 for Opus and 0.711 for Astra. Both models outperform MedSigLIP on 5 of 6 tasks while using more than 10,000 times fewer parameters. However, the same network trained with Adam on only 256 labelled images achieves a higher AUC on every MedMNIST task (mean AUC 0.828). On SynthUterus, Opus attains an AUC of 0.764, whereas Astra falls below chance (AUC 0.136) owing to a systematic error. These results show that LLMs can write functional medical classifiers, but that they do not yet replace training data. We release the protocol, logs and all generated weights at https://github.com/MaximilianLindholz/weight-space-programming-medical.