Task-Conditional Faithfulness Auditing of Multimodal LLMs for Grid Diagnosis
2026-07-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors present a method to check if multimodal large language models (LLMs) actually use the right kind of information when diagnosing power grids. Their method compares what the models say they rely on, how their behavior changes when some data is removed, and what engineers expect. If the models don't use the right evidence, the method fixes this by forcing correct evidence use and checks if the fix works without hurting performance. They tested this approach on different sizes of LLMs with standard power grid cases and showed it can find and correct problems in the model’s reasoning.
multimodal large language modelsgrid diagnosisfaithfulness auditself-reported reliancebehavioral reliancemodality ablationevidence-gated correctionIEEE 39-bus systemIEEE 118-bus system
Authors
Tianqiao Zhao, Meng Yue, Jianhui Wang
Abstract
Multimodal large language models (LLMs) can combine topology, measurements, and incident text for grid diagnosis, yet answer accuracy does not establish that task-appropriate evidence was used. This letter proposes a general framework in order to conduct task-conditional faithfulness audit. It compares self-reported reliance, intervention-derived behavioral reliance, and preregistered engineering importance. The framework first registers task-specific evidence requirements and compares them with self-reported reliance and behavioral changes under controlled modality ablations. To resolve detected discrepancies, we design an evidence-gated correction and re-audit mechanism that regenerates failed responses under evidence constraints and independently re-ablates them to verify improved grounding without performance loss. Case studies evaluate three differently scaled LLMs on IEEE 39- and 118-bus scenarios. These results validate the framework ability to detect, diagnose, and correct task-conditional faithfulness failures.