LLM explainers often fail to flag errors in autonomous agent reasoning
Triggers and Diagnostics for LLM-Based Interpretability Failures in Active Inference Agents
Machine LearningArtificial Intelligence
Summary
This paper shows that popular language models used to explain the decisions of smart computer agents often give fluent but incorrect explanations. When the agent’s inputs are tampered with or it makes wrong choices, the explanations usually do not warn the human operator. Instead, they provide smooth justifications that can even be manipulated by attackers. The authors suggest ways to fix these problems but highlight the need to test these explainers carefully before trusting them in real situations.
What this means in practice
- •For ai safety teams: Design evaluation procedures to detect silent failures and manipulation in LLM-based explanations of autonomous systems before deployment.
- •For grid operators: Use improved explainability audits to ensure trustworthy explanations from AI controlling grid demand during disruptions or adversarial inputs.
Authors
Param Raval, Rohit Shenoy, Archana Vaidheeswaran
Abstract
LLM explainers are increasingly attached to autonomous agents as runtime oversight, with operators reading a generated account of the agent's beliefs and actions rather than its internal state. We audit the account itself, pairing an Active Inference (AIF) agent that tracks German grid demand and adjusts generation with an LLM explainer on three backends (GPT-4o, Claude-3-Opus, Gemini), and probing the pair with three black-box triggers. Corrupting the observation stream by 600 MW per step moves the agent's posterior by 490 MW, roughly 0.9% of grid capacity. None of the 30 explanations produced during the injection flag anything under a stated rubric, and each narrates the corrupted belief fluently. On timesteps where the agent takes an objectively wrong action, all three explainers produce a sycophantic rationalization 80-95% of the time (n = 20 per backend). Attacker-controlled text in the observation metadata field steers the explainer, with susceptibility differing by provider and data exfiltration succeeding on all three. We propose mitigations for each failure but do not evaluate them. In every failure we observed, the explanation was fluent and wrong. Moreover, nothing in the explainer architecture checks whether an explanation is true before an operator acts on it. Testing the explainer therefore belongs in any audit of an agentic deployment.