Language models need many reasoning tries to find key decision points
Can LLMs Anticipate Their Cruxes?
Computation and LanguageMachine Learning
Summary
When language models help us make decisions, it's hard to understand the main reasons behind their choices because they produce so much text. The authors study how to find the most important sub-questions, or "cruxes," that influence the model’s decision. They compare three ways: asking the model beforehand, checking a single explanation, and looking at many explanations. Their tests show that only examining many explanations can reliably find these key points, especially when decisions are important.
What this means in practice
- •For ai system developers: Improve explanations from language models by collecting and comparing multiple reasoning outputs to identify key decision factors.
- •For risk assessment teams: Use extensive model rollouts to reveal critical uncertainties in AI recommendations for high-stakes decisions.
Authors
David Reber, James Reber, Ari Holtzman, Victor Veitch
Abstract
To use language models to help us make decisions we would like to understand the rationale behind the language model's outputs. In principle, the language model provides its rationale as part of its text output. However, the sheer volume of text makes this hard to use in practice. In this paper, we are interested in extracting the cruxes that lead a model to recommend one action over another. We take a crux to be a sub-question that a model's rollouts disagree on, where changing its value may have altered the conclusion. A crux is therefore a property of the distribution of a model's rollouts, not of any one rollout. We consider three strategies of increasing computational cost. First, we just ask a model to identify, a priori, what the cruxes of a decision will be. Second, we have the model actually go through the process of generating a response and then extract the cruxes that arose in its reasoning. Finally, we generate many responses and extract the cruxes by comparing the reasoning across the rollouts. We make this last procedure computationally tractable with cheap calls to a reranker, an estimated 86 times less compute than a judge. We then conduct an extensive empirical evaluation to assess whether the computationally inexpensive options are sufficient. They are not. Across domains and models, we find that a model's a priori assessment of the cruxes has little relation to the factors that are actually identified as important a posteriori. Using a single rollout helps, but is still inadequate for identifying the full set of cruxes. Accordingly, for high-stakes decisions (and to understand the space of language model decision making) the many rollout procedure is unavoidable.