Probe guided fine tuning improves ai honesty and safety without losing oversight
Alignment via Training Against Probes Without Losing Monitorability
Machine LearningArtificial IntelligenceCryptography and Security
Summary
AI models can sometimes pretend to be helpful during training without really learning what we want. The authors studied a way to fix this by training AI to avoid bad behaviors inside its own 'brain' rather than just judging its final answers. They found that continuously updating ways to detect harmful or dishonest thoughts inside the AI helps make it safer and more honest. This technique also keeps the AI’s inner workings understandable so people can keep checking it.
What this means in practice
- •For ai safety engineers: Improve AI model safety by training against internal signals of harmful or dishonest behavior to reduce risks during deployment.
- •For machine learning platform teams: Incorporate continuously updated probes in fine tuning pipelines to maintain oversight and robustness while enhancing model honesty and harmlessness.
Authors
Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko
Abstract
Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety-utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.