Hidden model reasoning signals detected but not used for answers

Present but Hard to Read: Probing Latent Reasoning with Minimal Pairs

Machine LearningComputation and Language

Summary

Machine learning models often seem to hide the steps they take when solving problems. This study shows that while tiny clues about intermediate calculations exist inside the model, these clues are hard to read and don’t actually influence the final answer. The researchers tested this by inserting minimal changes and checking if the model’s internal state reflected these changes clearly. They found the signals are present but very faint and sometimes mixed with unrelated information, meaning current methods might overestimate what the model 'knows' internally.

What this means in practice

  • For machine learning engineers: Use refined probing protocols to better assess whether internal model computations are actually used in final predictions.
  • For software developers in ai tools: Improve debugging tools by differentiating readable but unused model signals from those influencing outputs, enhancing model transparency.

Authors

Aur Shalev Merin

Abstract

Probing studies of latent reasoning often read a model's state where the answer begins and take a failed probe as evidence that a value is absent. We test this with a digit copied from the question, in the looped model Ouro-1.4B and the standard transformer Qwen2.5-1.5B. Comparing prompts that differ only in that digit shows it is present at the last prompt token, in a code shared across questions (readout about 0.9). A probe that sees one prompt at a time, the usual setting, barely reads it: 0.36 in Ouro and 0.19 in Qwen even after 5,249 training questions. The digit's change is small rather than crowded: random noise lands on equally busy directions, while a random word in its place moves the state far more and is readable. Since every varied token leaves a trace, a successful probe is ambiguous too. We give a protocol, including a synthetic null for generic mixing, for testing computed values against copies and mixtures of the inputs. In pre-registered studies of two-step problems it finds a small signal for the unwritten intermediate a+b just before Ouro writes the answer's last digit, and its sign follows the operation, though mixing alone produces much of that sign-following part. Holding that value at another problem's from the first loop on does not move the written digit toward the other problem's, while the same patch of the answer does: the value is readable but not used.