Large language model failures revealed through internal reasoning states

Uncovering Why LLMs Fail: From Internal Mechanisms to Adversarial Reasoning Trajectories

Artificial Intelligence

Summary

Large language models (LLMs) sometimes give wrong answers, but just seeing the final error doesn’t explain why they fail. The authors created a tool called CauTrace to look inside these models and track how their reasoning fails step by step. By defining different mental 'states' and measuring how the models change during reasoning, they found patterns where models struggle, especially with verifying information. Their approach helps explain failures in both normal and tricky test cases, offering clues for improving LLM safety and reliability.

What this means in practice

  • For llm developers: Diagnose and fix reasoning failures in language models by analyzing support, commitment, and verification processes during inference.
  • For security teams: Detect and understand adversarial attacks on language models by tracing how perturbations disrupt internal reasoning states.

Authors

Zirui Ding, Xinxin Fan, Hao Yuan, Shaoye Luo, Men Niu, Yunfeng Lu

Abstract

Large language models (LLMs) are usually evaluated by final-output correctness or safety, yet an endpoint label does not reveal how a failure develops throughout the reasoning process. We therefore ask whether ordinary reasoning failures and adversarially induced failures can be characterized using the same internal coordinates. Towards this end, we in this paper study a white-box diagnostic framework CauTrace to disclose the root-cause of why LLM fails from three fundamental inspections: Functional States, Propagation of Representation, Temporal-State Coordination and Reasoning Trajectory. First, seven functional states are creatively defined to represent the support and constraint processing in LLMs. Second, the propagation of representation change measures how the changes among the seven states persist across subsequent Transformer blocks. Third, temporal-state coordination and trajectory topology characterize when the commitment and verification occur relative to the support, and how the dominant state sequence evolves over the full inference process. Finally, we evaluate CauTrace on both clean reasoning failures and paired adversarial perturbations. Across Qwen and LLaMA-family models, the failed traces show weaker Attention-MLP agreement on support-state updates and less verification activity relative to commitment, while the exact representation-level signatures and timing shifts vary by architecture. Held-out repair and steering tests demonstrate consistent operational signals that were in line with our claims for certain models and specific settings, with aligned steering producing the predicted directional changes in process measures. Overall, CauTrace provides a unified diagnostic framework for analyzing failure dynamics through changes in support, commitment, and verification, which would shed light on the in-depth research on LLM security.