Chain-of-thought monitorability in AI models can be improved after training
Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability
Machine Learning
Summary
AI models sometimes explain their thinking in steps called chain-of-thought, which helps people check if the AI is reasoning safely and correctly. However, these explanations can become less reliable due to how the AI was trained. The authors propose a way to improve the reliability of these explanations after the main training is done, by encouraging the AI to make its explanations more strongly connected to its actual decisions. This approach successfully restores and even improves the ability to monitor AI reasoning in several tested models.
What this means in practice
- •For ai safety teams: Improve oversight of AI decisions by enhancing the trustworthiness of their reasoning explanations after initial training.
- •For enterprise ai developers: Increase the reliability of AI-generated explanations to better predict and prevent unsafe behaviors in deployed systems.
Authors
Usman Anwar, Tim Bakker, Christos Louizos, David Krueger
Abstract
Chain-of-thought (CoT) monitoring is one of the few practical tools available for overseeing capable AI systems, but it is fragile: a model's reasoning is only useful to a monitor if it remains coupled to what the model actually does. This coupling can be degraded during training through, for example, CoT compression, optimization against a monitor, and out-of-context generalization from synthetic documents. In this work, we propose an unsupervised post-training objective based on maximizing mutual information between CoT and the output for improving CoT monitorability. Across three model organisms with degraded CoT, it recovers most or all of the lost monitorability. Applied to undegraded policies, it further improves judged monitorability, improves prediction of unsafe behavior by CoT monitors and activation probes, and increases the prevalence of consequential reasoning. These results suggest that monitorability should be treated as a trainable attribute of the policy. A dedicated monitorability post-training stage can extend current practices that protect CoT from direct optimization pressure.