Speculative Probing: LLM Monitoring at Speculative-Decoding Cost
Artificial IntelligenceComputation and Language
Summary
The authors found a new way to quickly classify language model outputs during their own processing without slowing things down much. Instead of using slow, separate classifier models or simple checks that aren’t very aware of context, they reuse a part of the model’s decoding process by adding a small trained prompt to the text. This makes classification efficient and accurate, as tested on several tasks and models, sometimes even matching specialized safety classifiers. Their method avoids the usual trade-off between speed and accuracy in real-time classification.
Authors
Collin Zhang, Tingwei Zhang, Vitaly Shmatikov
Abstract
Real-time classification during language model inference is valuable for safety filtering, behavioral analysis, and model monitoring, but current approaches force a trade-off between accuracy and efficiency. Hidden-state probes are fast but limited: they are either not context-aware: operating on a single vector and cannot model interactions across positions; or they are very costly: having dedicated classifier models (Llama Guard, Qwen Guard, LLM-as-judge) or performing computation on hidden states for all tokens and then pooling the results (MultiMax). This shows an intrinsic trade-off between efficiency and accuracy. However, we find that the speculative-decoding module in recent LLMs can be repurposed for efficient high-quality classification. By appending a trained soft prompt at the end of the target sequence, we can repurpose the speculative-decoding module into a sequence classifier. At inference time in a speculative-decoding pipeline, the KV cache is already in GPU memory, so classification adds negligible overhead. We evaluate on four classification tasks across four models (Qwen3.5-4B, 9B, 27B, MiniCPM4.1-8B). Our small probes consistently outperform zero-shot GPT-5.4-mini and, on multilingual prompt safety, match or beat specialized 8B safety classifiers (Qwen3Guard-Gen-8B, Llama-Guard-3-8B) without running a full LLM.