Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
2026-07-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors identify a problem in automatic prompt optimization (APO) for vision-language models (VLMs): these optimizers can't see the images when fixing errors, so they miss important visual details. They propose Cross-Modal Visual Feedback (CMVF), where a stronger model looks at the images that caused mistakes without knowing the answers, helping find visual error patterns. This info is then used to improve prompts, but the final system still only needs text input, keeping efficiency. Their tests show CMVF improves performance across many datasets and models, and it naturally creates reusable visual checklists that work across different models.
Automatic Prompt OptimizationVision-Language ModelsCross-Modal Visual FeedbackVisual Question AnsweringMultimodal TasksModel OptimizationError DiagnosisPrompt EngineeringInference Cost
Authors
Haoyue Liu, Xiaoyu Ma, Ye Chen, Yuexian Zou, Xiaoying Tang
Abstract
Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results. However, on multimodal tasks, the effectiveness of APO is fundamentally bottlenecked by a blind feedback channel: the optimizer reads the question, the prediction, and the gold answer, but never the input image on which the model failed, and therefore cannot diagnose visually grounded errors. As a remedy, we introduce Cross-Modal Visual Feedback (CMVF). CMVF incorporates (1) a failure-conditioned visual diagnosis stage, in which a stronger optimizer VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns that drive the prompt rewrite. Crucially, the image is consumed only during optimization; the deployed artifact is an ordinary text prompt that runs at the same inference cost as any text-only baseline. Extensive results across 12 VQA datasets and 4 target VLMs demonstrate that CMVF consistently ranks first, improving over the strongest baseline on every target by 2.4 points on average, with gains of up to 6.5 points on individual benchmarks. Moreover, the optimizer self-organizes into expert-style visual checklists that transfer across models without re-optimization.