Activation Probes Surface Code-Security Signals that the Model's Output Misses
2026-08-10 • Cryptography and Security
Cryptography and SecurityMachine Learning
AI summaryⓘ
The authors studied AI tools that write code and how to detect security problems in that code. They found that by analyzing internal signals (called activations) from an open AI model reviewing the code, they could better spot security issues than by just asking the model directly. They trained a simple classifier on pairs of flawed and fixed code and then tested it on new security problems the model hadn’t seen before. Their method worked better at identifying vulnerable code than just reading the model’s yes/no answers or explanations. This shows looking inside the AI's reasoning process can reveal security warnings that straightforward questions miss.
AI coding agentssecurity reviewmodel activationslinear probevulnerability detectionopen-weight modelscode securitylogitschain-of-thought promptingPython functions
Authors
Ivan Wiryadi
Abstract
AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated. The agents in widest use are closed-weight, so a deploying team cannot read their internals. It can instead run an open-weight model as a reviewer over the agent's output. That reviewer's activations are readable. We ask whether reading those activations recovers a security signal that simply asking the same reviewer misses. We fit a single linear probe per model on a corpus of paired vulnerable-and-fixed Python functions, then test it without retraining on real disclosed vulnerabilities whose weakness type the probe never saw in training, across five open-weight reviewer models. On the vulnerabilities fixed by changing a single function, the probe scores the vulnerable function above its fix on 61-67% of cases for every model, beating the 50% chance line. It also beats the same model's prompted YES/NO win-rate read from its logits, under every prompt we try. Asking the model for a written verdict, even with chain-of-thought, returns the same answer on the vulnerable and fixed function most of the time and so cannot tell them apart. Model activations carry a code-security signal that prompting the same model misses.