Verbalising model reasoning improves chain of thought faithfulness detection
Verbalising Model's Computation for Chain-of-Thought Faithfulness Detection
Computation and Language
Summary
Large language models sometimes produce reasoning steps that don’t fully match how they actually think. This paper introduces VERITAS, a way to turn the model’s internal calculations into readable explanations at each step of its reasoning. By looking at these explanations, the authors train a classifier that better detects when the model’s reasoning goes off track compared to previous methods. VERITAS also helps understand why and where the model deviates, making it easier to trust or debug the reasoning.
What this means in practice
- •For machine learning engineers: Detect when language models produce unfaithful reasoning to improve trustworthiness in AI applications.
- •For ai system auditors: Gain better understanding of where and how model reasoning diverges to support explainability in AI audits.
Authors
Yujing Wang, Yulan He, Wenya Wang, Hanqi Yan
Abstract
Current Chain-of-Thought (CoT) verification methods detect reasoning faithfulness from model outputs or static activations, but are limited in explaining how unfaithfulness develops during generation. We introduce VERITAS to verbalise the model's computation: how different reasoning anchors emerge, transfer across layers, and surface in the output. VERITAS adopts the layer-wise Jacobian projection to produce a verbalised account of the model's internal reasoning evolution at each generated CoT token. By training a classifier on the extracted features along the traces, we show that these structured features provide insightful perspectives on how and where reasoning diverges toward unfaithfulness. Experiments on four datasets from FaithCoT-Bench and BonaFide show that VERITAS achieves a significant performance gain over LLM-as-Judge, with a 28.4% gain in balanced accuracy, and an average improvement of 16.0% over activation probing. Further analysis shows VERITAS can be generalised to different datasets and model architectures. Our work shows that verbalising model computation provides a richer and more interpretable lens on CoT faithfulness, moving beyond detection toward a mechanistic understanding of how reasoning fails.