MemeMind: Reference-Guided Trace Construction for Offline Context Optimization
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors propose MemeMind, a method that improves an AI's understanding of complex memes by adding missing successful examples when normal attempts fail. They use a tool called TraceBuilder to find and verify evidence supporting the correct answers, and ToolGuide to create helpful instructions for the AI tools. This process only happens during training, while the AI model itself remains unchanged. They test MemeMind on a challenging meme interpretation benchmark and find it significantly improves performance compared to previous methods. The biggest improvement comes from creating successful examples for situations where the AI initially failed.
offline context optimizationrolloutsvisual groundingimage retrievaltext searchmeme interpretationadaptation bufferQwen3-VL modelsevidence acquisitionbenchmark evaluation
Authors
Run Yang, Weihang Wang, Boheng Sheng, Yuchen He, Jielei Zhang, Pengyu Chen, Zhiyu Wu, Qiang Sun, Huyang Sun, Longwen Gao
Abstract
Offline context optimization improves an agent by revising its instructions and examples while keeping the model frozen. This approach learns from rollouts on an adaptation set, but some queries produce only failed rollouts. In these cases, the optimizer sees no successful example of how the available tools can reach the correct answer. We introduce MemeMind, which uses an offline reference answer to recover this missing experience. TraceBuilder identifies the evidence required by the reference, executes text search, image retrieval, and visual grounding, and verifies the resulting tool trace before adding it to the adaptation buffer. ToolGuide then summarizes the collected traces into a shared guide and separate instructions for each tool. The reference answers and constructed traces are used only during adaptation, while inference uses the learned guides with a frozen model. We study this problem through Anime, Comic, and Game meme interpretation. These memes combine edited and ambiguous visual content, overlaid text, long tail franchise knowledge, and culture specific references. Their interpretation can require coordinated visual grounding, image retrieval, and text search, making them a demanding setting in which native rollout groups may fail together. We evaluate MemeMind on MemeX, a benchmark of 1,000 such memes annotated by experts. Across two Qwen3-VL models, two language partitions, and two independent judges, MemeMind improves over the strongest context optimization baseline by 22.0% and 21.1% on Qwen3-VL-30B-A3B, and by 8.1% and 8.0% on Qwen3-VL-235B-A22B under GPT-5 judging. Ablations and held out traces show that constructing successful tool use for failed groups provides the largest component gain and produces more effective evidence acquisition at inference time.