Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments
2026-08-17 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors propose a new way to test if explanations for AI model behaviors are useful by seeing if these explanations help predict how models act when given slightly changed inputs. They create a system called CHIVE that finds surprising model responses and makes small edits to prompts to understand why those responses happen. Using CHIVE, they discover many clear explanations and see that common explanation methods do not help predict model changes better. They also show that training on CHIVE's explanations helps models do better on new types of problems. Overall, their work helps find and check explanations for language model behaviors automatically.
counterfactual simulatabilitylanguage modelsinterpretabilitychain of thoughtprompt editingmodel behaviorexplanation evaluationtraining data generationout-of-distribution generalizationCHIVE
Authors
Adam Karvonen, Euan Ong, Subhash Kantamneni, Samuel Marks
Abstract
Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.