Multi-image model failures traced to identifying image instances
What Breaks in Multi-Image Schema Application? Factorial Oracle Interventions Reveal an Instance-Information Bottleneck
Computer Vision and Pattern Recognition
Summary
Some vision-language AI models look at many images and answer questions about them, but it can be hard to know where they go wrong. The authors studied a task where a model had to find two characters in a series of pictures and name them based on their looks. They tested the model with helpers that gave it the exact character information or the exact character descriptions. The biggest improvement came from giving the model the exact descriptions of the characters’ appearances, showing that the main problem is recognizing and ordering those characters before using the rules to name them.
What this means in practice
- •For computer vision engineers: Improve multi-image reasoning models by focusing on instance identification before rule-based schema application.
- •For ai testing teams: Use oracle-based factorial interventions to diagnose where multi-image vision-language models fail.
Authors
Yiliu Xu, Xinjin Li, Yu Ma
Abstract
Modern vision-language models can accept multi-image contexts, but aggregate accuracy rarely reveals where a multi-image reasoning pipeline fails. We study a controlled schema-application task in which a model must locate two target characters in a 16-image sequence, identify their visual attributes, consult an item-specific visual codebook, and compose two ordered nonce code names. We introduce a within-item 2-by-2 factorial Oracle design: Oracle-A supplies a verified textual schema, Oracle-B supplies a verified ordered target-appearance representation, and both interventions retain the original images and final question. Across 72 new items and 1,188 analyzed model responses, Oracle-B increased strict exact accuracy by 54.6 percentage points (95% confidence interval: 46.3 to 63.0) and outperformed Oracle-A by 60.6 points (Holm-adjusted p-value no greater than .0002). An auxiliary retrieval diagnostic reached only 49.1% ordered-appearance accuracy even after removing code-name generation. Providing both Oracle sources yielded 100% accuracy across all models. These results localize the dominant recoverable deficit to upstream ordered instance recovery rather than primarily to downstream schema use and composition.