Concept based models can worsen predictions when only partly fixed
Harmful Partial Interventions and Benign Leakage in Concept-based Models
Machine LearningArtificial Intelligence
Summary
Concept-based AI models let people check and fix parts of how the model thinks, using human-understandable ideas. But the authors show that trying to fix only some parts can sometimes make the model's overall predictions worse, which they call intervention harm. They prove that previous ideas on when fixing should help are not enough and find that missing or incomplete concept information increases the risk of problems. They introduce a new training method that reduces these harmful effects when fixes happen partly, without hurting how well the model predicts.
What this means in practice
- •For machine learning engineers: Improve concept-based AI systems in safety-critical applications by reducing harmful effects of partial corrections during human interventions.
- •For medical ai developers: Enhance reliability of AI models that allow doctors to intervene on intermediate concepts without risking unintended prediction damage.
Authors
Gregory Verghese, Christopher J. Soelistyo, Mateo Espinosa Zarlenga, Alessandro Barp, Christopher R. S. Banerji
Abstract
Artificial intelligence systems deployed in high-risk settings increasingly require models that support meaningful human oversight. Concept-based models (CMs) offer a promising framework by enabling users to inspect and intervene on intermediate, human-understandable concepts. However, concept interventions can counterintuitively worsen downstream predictions, a failure mode we formalise as intervention harm. We distinguish harm after all concepts have been intervened on from harm under partial interventions, the more likely practical use case. Modern CMs often achieve high accuracy by encoding, or leaking, additional task-relevant information in their intermediate concept representations. Recent work argues that such leakage can be benign with respect to intervention harm when concept representations satisfy two conditions: sufficiency and localisation. We prove that these criteria are not sufficient to guarantee that concept interventions improve downstream performance. Across controlled and real-world settings, we find that concept supervision incompleteness is associated with increased intervention harm. Importantly, optimising the model when all concepts are intervened on, a previously proposed leakage mitigation objective, reduces endpoint harm but does not reliably prevent harmful partial corrections. Hence, we introduce a complementary monotonic intervention objective that directly penalises increases in task loss along partial intervention trajectories. Combining endpoint and monotonic intervention training substantially reduces the prevalence and magnitude of partial intervention harm while preserving concept and task predictive performance.