Visual language models learn better by using their own mistakes

The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Choosing good example images helps vision-language AI make better predictions, but it's usually complicated to find these examples. The authors found that the AI’s own mistakes are actually useful because they show where it tends to go wrong. Their method collects these mistakes and then uses them to improve answers without extra complicated steps. This approach works well on different tests and models, making it both faster and more accurate than previous methods.

What this means in practice

  • For computer vision engineers: Improve accuracy and speed of multi-task vision-language systems by integrating model-specific error examples for on-the-fly adaptation.
  • For customer support automation teams: Enhance visual question answering capabilities in chatbots by using the method to reduce frequent mistakes in interpreting images.

Authors

Shu Zhao, Tan Yu

Abstract

Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/