Arbor tracks reasoning states to pinpoint errors in long problem solving

ARBOR: Locating Reasoning Failures in Long Chains of Thought through Reasoning State Tracking

Computation and LanguageArtificial Intelligence

Summary

Many large AI reasoning models solve problems by thinking through many steps, but sometimes they make mistakes due to thinking too much, too little, or making unsupported jumps. The authors introduce ARBOR, a new way to watch what each step adds to the ongoing reasoning and see where mistakes happen by tracking how the reasoning changes over time. This method helps break down failures into clearer types and tells which steps are truly wrong versus ones likely to fail. This tracking can improve how AI systems fix or avoid errors while solving hard problems.

What this means in practice

  • For ai model developers: Identify exactly which reasoning steps cause errors in AI problem-solving chains to improve debugging and training.
  • For automated reasoning engineers: Use tracked reasoning states to guide real-time corrections or halting before reasoning failures occur in complex AI tasks.

Authors

Huihan Li, Siyuan Wang, Naga Vamsi Ramana Dinavahi, Xiaohui Liu, Xiang Ren

Abstract

Large reasoning models solve complex problems through long chains of thought, in which prior work has documented failure modes such as overthinking, underthinking, and unsupported deductions. Existing methods detect these failures from local signals: textual similarity to nearby steps, switch tokens, or an LLM's judgment of a step's correctness. However, what a step contributes to the solution depends on the state that earlier steps have established, so identifying failures requires a global view that tracks that state across the trajectory. Therefore, we propose ARBOR, a reasoning-failure attribution framework that represents each trajectory as a tree of reasoning states, built as each step creates, verifies, prunes, or revisits a state, and attributes each failure to a relation between a step and the state preceding it. Using ARBOR, we refine the three failure modes into distinct subtypes that harm reasoning through different mechanisms and at different costs. We find that the tree state ARBOR maintains before a step separates failure steps from the at-risk steps at which the failure did not occur. We discuss how ARBOR integrates with existing mitigations at inference time and in training data. Code, data and interactive examples are available at https://inklab.usc.edu/ARBOR.