From Plausible to Actionable: A Position on LLM Self-Explanations
Computation and Language
Summary
The authors discuss how Large Language Models (LLMs) can explain their own decisions, called self-explanations. They point out that while these explanations often sound believable, it's unclear if they truly reflect the model's real thought process. The authors highlight that typical ways to check these explanations have limitations and suggest better methods to judge how believable and accurate they are. They also emphasize that these explanations can be useful in helping people make decisions, even if they’re not perfectly faithful to the model’s reasoning.
Authors
Elize Herrewijnen, Benedetta Muscato, Gizem Gezici, Fosca Giannotti
Abstract
Large Language Models (LLMs) can generate natural language explanations that rationalize their own decisions, a phenomenon commonly referred to as self-explanations.Such explanations have emerged as a promising direction for explainable artificial intelligence (XAI), particularly for interpreting LLM behavior.However, while self-explanations often appear plausible, whether they faithfully reflect a model's underlying reasoning process remains an open question. In this opinion paper, we argue that self-explanations can be highly plausible, questionably faithful, and yet highly actionable. From a traditional XAI perspective, we identify the limitations of standard evaluation protocols for LLM-generated self-explanations and propose practical guidelines for assessing their plausibility and faithfulness. Moreover, we argue that evaluation should extend beyond these criteria to actionability, highlighting applications of LLM rationalization capabilities that support informed decision-making and appropriate action across diverse stakeholders.