Chain of thought reasoning must leak information but not always be decoded
Hidden Reasoning Must Leak, but Need Not Be Readable: Fundamental Opportunities and Limits for Chain-of-Thought Monitoring
Machine LearningArtificial IntelligenceComputation and Language
Summary
This paper examines whether AI models that explain their reasoning step-by-step can hide some calculations from anyone checking their work. The authors find that simple hidden computations can be performed without revealing much, but as tasks get harder, the process inevitably leaks lots of information. However, this leaked information might be hidden in a way that makes it unreadable to monitors, much like encrypted messages. These results reveal both the limits of detecting hidden reasoning and how some AI systems could secretly carry out computations without easy detection.
What this means in practice
- •For ai safety teams: Determine when reasoning traces from models must reveal hidden internal tasks based on their complexity and size.
- •For security engineers: Design monitoring tools aware that hidden computations may be encrypted and unreadable even if detected in output.
A theory result. No direct application yet.
Authors
Mohammadali Mohammadkhani, Madhava Krishna, Yash Sarrof, Michael Hahn
Abstract
Can reasoning models trick chain of thought (CoT) monitors and perform hidden computation without revealing it in their thinking traces? We show that the answer depends on the underlying task difficulty and the model size. Simple computations can be performed covertly; however, beyond a threshold depending on model size, successfully solving the task necessarily leaks a near-linear amount of information about the covert task input into the CoT. Therefore, sufficiently complex hidden computation always leaves an information-theoretic footprint. However, concerningly, this leakage need not be readable: Under plausible cryptographic assumptions, even a one-layer Transformer can encrypt its reasoning online so that no polynomial-time monitor can extract information about the hidden computation. Overall, our theoretical and empirical results provide a holistic view of both the opportunities and the limitations of CoT monitoring.