Causal task breakdown reveals where large models really fail
Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Large language and vision models sometimes make mistakes, but it’s hard to know if they fail because they lack a skill or because they were tripped up earlier in a task. The researchers created a way to break down complex tasks into smaller parts and check whether errors come from specific skills or earlier mistakes. They tested this with a big set of questions covering many areas like perception and reasoning. This approach helped find out that many errors happen because models didn’t get earlier steps right, especially for thinking tasks.
What this means in practice
- •For ml system developers: Identify which specific capabilities in complex tasks cause downstream errors to guide model improvement.
- •For ai quality assurance teams: Prioritize testing on critical prerequisites that cause most failures to improve model evaluation efficiency.
Authors
Xia Hu, Brian Potetz, Chun-Ta Lu, Huanfen Yao, Leonidas Guibas, Zhicheng Wang, Howard Zhou, Pengfei Xing, Andrew Gallagher
Abstract
End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54\% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84\% of the gain from supplying all prerequisites.