Vision language models keep clearer reasoning trails with targeted training
Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation Enforcement
Artificial Intelligence
Summary
Vision-language models can think through problems step-by-step, but reinforcement learning sometimes makes their reasoning less clear and harder to follow, even if they get the right answers. The authors studied what changes inside these models when this happens and found specific parts of the model’s internal activity that become less distinctive. They designed a method called TAME to gently reduce distracting patterns during training, which helps keep the reasoning steps more understandable to humans without hurting performance. This makes it easier to check and trust these models when they explain their answers.
What this means in practice
- •For ai safety engineers: Use TAME to improve transparency of reasoning steps in multimodal AI systems, aiding audit and oversight during model deployment.
- •For product teams deploying ai assistants: Enhance user trust by maintaining clearer chain-of-thought explanations in vision-language models through targeted training techniques.
Authors
Xutao Mao, Jianing Zhu, Jinman Zhao, Tongliang Liu, Xiaowen Chu, Cong Wang, Bo Han
Abstract
Reinforcement learning (RL) improves reasoning in vision-language models (VLMs) but can induce chain-of-thought (CoT) obfuscation: an operational, non-intentional outcome where task reward or accuracy rises while traces become less grounded and monitorable. Prior work largely documents this decay behaviorally, leaving its representation-level correlates and actionable controls unclear. We find that template- and ground-associated activations become less separable during RL; matched interventions support the contribution of selected features to monitorability degradation. Guided by this evidence, we propose Targeted Anti-obfuscation with Mechanistic Enforcement (TAME), which uses Sparse Autoencoders (SAEs) to combine behavioral feedback with targeted suppression of template-associated activations during RL. Its asymmetric constraint penalizes template activations only above their pre-RL baseline, anchoring the localized features while behavioral feedback promotes grounded refinements. Across VIRL-39k, SPA-VL, and two model families, TAME improves CoT monitorability by up to 30.9 and 16.7 percentage points over Group Relative Policy Optimization (GRPO), respectively. Blinded human evaluation finds higher human monitorability on both datasets, and two held-out monitor families reproduce the monitorability gains. Task accuracy changes are small and mixed, and general-capability benchmarks show task-specific trade-offs. These results provide a path from behavioral monitoring to representation-level oversight for more auditable RL-trained multimodal systems.