Multimodal in-context learning underperforms text only but can improve
TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
Computer Vision and Pattern RecognitionMachine Learning
Summary
Models that learn tasks by example usually do better with text instructions than with images. The authors created TwinICL, a new test set with matched text and images to compare how well models learn from each type. They found that models struggle more with images, but combining some techniques can boost performance on visual tasks. Even when told the task clearly, models still do worse with images, showing that image-based task learning is harder.
What this means in practice
- •For machine learning engineers: Use TwinICL to evaluate and improve how models learn from both visual and text examples.
- •For ai product developers: Assess multimodal models for user interfaces that involve both images and text to ensure balanced task performance.
Authors
Zihan Xue, Po-Yi Lu, Serhii Honcharenko, Zih-Ching Chen, Hsuan-Tien Lin, Nanyun Peng, I-Hung Hsu, Kuan-Hao Huang
Abstract
In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong multimodal ICL performance on a diagnostic subset, despite limited or inconsistent individual effects. To distinguish difficulties in executing tasks from those in inferring them, we evaluate models with explicit task instructions, revealing a modality gap even when the task is known. We then examine how adding demonstration inputs and outputs reshapes this gap, highlighting demonstrations' dual role as additional context to process and evidence about the task. The dataset is available at https://github.com/lab-flair/TwinICL.