Papers for

ml system developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Causal task breakdown reveals where large models really fail

Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis

Abstract: End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54\% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84\% of the gain from supplying all prerequisites.

Wed 30 SeptComputation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Large language and vision models sometimes make mistakes, but it’s hard to know if they fail because they lack a skill or because they were tripped up earlier in a task. The researchers created a way to break down complex tasks into smaller parts and check whether errors come from specific skills or earlier mistakes. They tested this with a big set of questions covering many areas like perception and reasoning. This approach helped find out that many errors happen because models didn’t get earlier steps right, especially for thinking tasks.
Open → 2609.38851v1

Gradient imbalance impacts vision language model fine tuning differently across domains

Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domains

Abstract: Imbalanced gradient magnitudes between the visual and language pathways of a vision-language model are often treated as a defect to be corrected. We test that premise for one family of correction, deliberately excluding adaptive, signal-driven schemes (e.g. BalGrad, OGM, PMR, CGGM), which are a mechanistically distinct class outside this study's scope. Measuring the language-to-visual gradient-norm ratio, reported in parameter-normalised form, across nine fine-tuning conditions, three seeds, and three captioning datasets spanning distinct physical domain shifts -- underwater, aerial, radiological -- we find imbalance magnitude varies markedly across domains with no predictable ordering. A plain learning-rate reduction cuts imbalance substantially and lands within a few BLEU points of the best method on every dataset. Staged freezing reduces the ratio on every domain yet never ranks first; a schedule-only control isolates freezing as the cause on one dataset but not the other two. Forcing the two gradient groups to equal magnitude drives per-parameter imbalance close to zero on every domain, yet is both the best result in the study and the worst placement among full fine-tuning methods, on different datasets, with identical settings. Reductions in gradient-norm ratio do not consistently predict captioning performance across domains, and how a given level of balance is reached matters as much as the level itself. As a secondary finding, a commonly reused LoRA configuration applied to BLIP silently adapts zero visual parameters; correcting it improves BLEU-4 on all three datasets.

Sun 20 SeptComputer Vision and Pattern Recognition
The gist
Training models that connect images and text can involve uneven updates between the visual and language parts, which people often try to fix. The authors studied whether balancing the sizes of these updates always improves performance across different image types like underwater or aerial photos. They found that how much imbalance there is and whether fixing it helps changes depending on the type of images used. Simple steps like lowering learning rates or gradually freezing parts affect imbalance differently, and forcing exact balance does not always result in better captions. They also found a common setup for fine-tuning was accidentally leaving some visual parts unchanged, and correcting this helped performance on all tested datasets.
Open → 2609.23655v1