Trace framework measures explainability issues in AI systems
TRACE: A Governance Framework for Measuring Explainability Debt in Production AI Systems
Machine LearningArtificial Intelligence
Summary
AI systems used in important areas like finance can gradually lose the ability to clearly explain their decisions when people ask for reasons. The authors propose TRACE, a set of tools to measure and track this problem, called Explainability Debt, over time. They tested it on a fraud detection system and showed how it can predict and locate where explainability problems build up, especially in high-risk cases. This helps organizations meet legal requirements for AI transparency and accountability.
What this means in practice
- •For financial compliance teams: Monitor and forecast AI explainability risks in fraud detection to ensure regulatory compliance.
- •For risk management professionals: Identify and prioritize AI decisions with low explainability confidence to better allocate review and auditing resources.
Authors
Harish Kant Pathak
Abstract
Production AI systems deployed in high-stakes domains accumulate a governance liability that existing monitoring frameworks fail to detect: the progressive inability to explain individual decisions when regulators, auditors, or affected individuals demand accountability. We introduce TRACE (Transparency, Risk, Accountability, Compliance, and Explainability), a seven-instrument governance framework for measuring, tracking, and remediating Explainability Debt in production AI systems. The foundational instrument, the Explainability Debt Score (EDS), quantifies the proportion of production decisions falling below a governance-defined explainability confidence threshold at any point in time. Complementary instruments include DART (Debt Accumulation Rate Tracker for breach forecasting), SHIV (Scenario Health and Integrity Validator for daily governance), FDE (Feature Drift Evaluator for causal attribution), HVE (Human Validation Engine), AIDE (Audit Intervention Decision Engine), and ZERO (Zero Explainability Risk Optimiser for remediation). Through a twelve-month longitudinal case study of a production fraud detection system processing 50,000 daily financial transactions, achieving 98.46% accuracy and ROC-AUC of 0.9990, we demonstrate that an EDS of 0.23 on audit day was statistically predictable six months in advance using DART trajectory analysis (beta = 0.008/week, R-squared = 0.94, 95% CI: [0.006, 0.010]), and that 78% of Explainability Debt was concentrated in the highest-regulatory-risk decision category (transactions above $10,000), a risk asymmetry completely invisible to system-level metrics. TRACE provides the first quantitative operational architecture for EU AI Act Article 13 compliance in production AI deployment, establishing a new subdiscipline of explanation governance distinct from explanation generation.