Calibrated readings let users measure neurons in ai models reliably
Calibrated concept profiles: making the reading of a model's neurons a measurement
Computation and LanguageMachine Learning
Summary
Understanding what individual neurons in AI models represent has been tricky because their activations change with different inputs, and there hasn’t been a consistent way to measure them. This paper introduces a method where the authors define a standard set of inputs to test neurons, turning their activations into meaningful, comparable numbers with known units and noise levels. By doing this repeatedly on a code-writing AI model during training, they show which neurons correspond to specific concepts over time. This approach helps make neuron interpretation more like a precise measurement that others can reproduce and compare.
What this means in practice
- •For ai model evaluators: Measure and compare individual neuron responses reliably across model checkpoints using calibrated profiles.
- •For machine learning engineers: Identify which neurons represent specific programming concepts during AI code model training for better interpretability.
Authors
Piotr Wilam
Abstract
Interpretability has largely moved from reading a model's neurons directly to reading a second model trained on the first: a probe, a sparse autoencoder, a crosscoder. The move answered real faults -- mixed meanings, thresholds without a unit, readings that did not transfer from one study to the next -- but gave up what a direct reading alone provides: nothing trained in between, the neurons themselves named, and a reading that can be repeated and compared. Here the direct reading is made a measurement. A neuron's activation varies with the prompt, so its reading is a distribution over a population of stimuli. We declare that population: every factor around a target construct is drawn at its rate in a reference corpus and logged. From the declared distribution we generate a battery of prompts, and draw it twice. From the battery we compute the calibrated concept profile, one number per neuron, stated in units of the neuron's own spread on ordinary input; the two draws give every number a noise unit. Before the profile is read, a fixed set of checks licenses the battery on the model, and the smallest change the profile can resolve across training checkpoints is measured. Realised for Python on OpenCoder-1.5B and followed across checkpoints spanning two trillion training tokens, the profile gives what the alternatives do not report: a unit on every number, a chance level for every comparison, and which neurons carry a construct at every checkpoint. The battery, the code and the data behind every result are released.