Explainable AI needs focus on models not just interpretation methods
From Interpretability Methods to Interpretable Models
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
People have created many tools to explain how AI systems for vision work, but most of the effort has been on making and comparing these tools rather than understanding the AI models themselves. The authors suggest shifting attention to examining what AI models actually represent and compute to see how understandable they really are. They emphasize that understanding must come from regular users or independent evaluators, not just AI experts. This approach could help build more trustworthy AI by measuring human understanding directly. They review existing tools and research, and propose a new agenda focused on interpretable models.
explainable AIinterpretabilitycomputer visionattribution methodsfeature visualizationconcept-based methodscircuit-based methodsmodel understandinghuman evaluationtrust in AI
Authors
Julien Colin, Nuria Oliver, Thomas Serre
Abstract
More than a decade in, explainable AI (XAI) for computer vision has assembled a mature toolbox: attribution, feature visualization, concept-based, and circuit-based methods. Yet almost all of the field's effort has gone into building and comparing these methods, and little into the question they were meant to answer---how interpretable are our models, and are we making progress as they evolve? We argue for shifting the field's focus from methods to models, along two complementary lines. One is already within reach: existing tools let us characterize and compare what different models represent and compute. The other is harder, and largely neglected: whether a model can actually be understood by the humans who rely on it---the independent evaluators on whom trust and certification depend, not the experts confirming what they already expect. It can only be measured, not inferred. We review why the toolbox is mature enough to support both, survey the thin body of work comparing models, draw a parallel to systems neuroscience, and close with a model-centric XAI agenda.