Summary
Many methods designed to explain how artificial intelligence models work focus too much on scoring systems that may not fully capture the quality of these explanations. The authors look closely at one popular method called GAE, which tries to explain how a model pays attention to different parts of input data, and find that it has some problems. They propose a new approach named CAH that combines different ideas to give clearer, more reliable explanations, especially for models with special tokens. This new method draws inspiration from concepts in economics and may help improve future ways of understanding AI models.
explainable artificial intelligencetransformer modelsattention mechanismGAECAHmodel interpretabilityproxy metricsfeature-based methodsprocess-based methodszero-sum games
Abstract
We observe a phenomenon that current algorithmic research in the field of explainable artificial intelligence primarily pursues better performance on several proxy metrics. On the one hand, these proxy metrics themselves are more or less flawed and cannot properly measure the quality of methods. On the other hand, metric-oriented research approaches often lead to the neglect of the rationality and interpretability of the methods themselves. Explainable artificial intelligence is abbreviated as XAI. The metric-driven research paradigm has resulted in a lack of interpretability of the relevant XAI methods themselves. Accordingly, there is a need for interpretability research on XAI methods, which can be playfully referred to as XXAI. This paper is one of our works on XXAI. This paper takes Generic Attention-model Explainability (GAE), a widely influential model interpretation method , or rather, XAI method that represents an important technical route, as the research object, and explores the real working mechanism and flaws of this method as well as the technical route it represents. Based on the conclusions of this study, it may be necessary to re-examine or verify GAE-related methods and their domain applications. We argue that GAE is an interpretation method that focuses on the attention process. After pointing out the working mechanism and flaws of GAE, we propose Cumulative Asset Holdings (CAH), a more reasonable Transformer interpretation method integrating both process-based and feature-based ideas from an economic zero-sum games perspective. In addition, it is worth noting that our method is applicable to models with special tokens, where existing methods may suffer from limitations. The model simplification research method and the analysis of additive operations adopted in this study may provide inspiration for other research works in XAI.