How audit rules influence truthful factor explanations in large language models
How the Audit Rule Shapes Faithful Factor Explanations in LLMs
Computation and Language
Summary
Sometimes large language models are asked to explain which parts of their input influenced their answers. Checking these explanations takes time and resources, so only some parts are verified. The authors studied how this limited checking changes the model’s incentive to be honest. They found that if the audit depends on what the model reports, it might hide some important factors to avoid being audited. Adding a small amount of random, independent auditing encourages honesty. They tested this idea using scores and real language models and found that it works as expected.
What this means in practice
- •For ai product teams: Design explanation tools that include some random, report-independent audits to encourage honest reporting of important input factors in language models.
- •For model evaluation teams: Use mixed auditing rules combining report-dependent and independent checks to better verify factor-level explanations under limited verification budgets.
Authors
Taolin Zhang, Hanyu Wang, Jiuheng Wan, Tingyuan Hu, Chengyu Wang
Abstract
Large language models are often asked which input factors influenced their outputs. For structured inputs, such reports can be checked by counterfactual perturbation, but each factor must be queried multiple times to estimate its effect, so verification is usually budget-limited. We study how this limited-budget setting changes the incentive to report factor-level influence truthfully. We formalize the interaction as a verification game and show that proper scoring alone is not enough when auditing depends on the report: report-dependent auditing creates a suppression incentive, because factors reported as important are more likely to be checked and penalized for estimation noise. In contrast, report-independent auditing, or a mixed rule with a small report-independent floor, removes this channel and makes truthful reporting preferable to full suppression. We instantiate the framework with the Counterfactual Brier Score (CBS) and evaluate its predictions on four NLP benchmarks. A synthetic rational agent matches the theoretical prediction exactly, and real LLMs follow the same incentives when they are made explicit. The main design implication is simple: under partial verification, factor-level explanation systems should include a report-independent audit component so that under-reporting cannot be used to avoid scrutiny.