New dataset creates image based instructions for reasoning tasks
Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission
Computer Vision and Pattern Recognition
Summary
Many AI systems learn better when they have lots of examples showing how to understand images and follow instructions together. The authors made a new dataset by creating short, clear instructions based on images from an existing collection. They used AI models to write, check, and pick good instructions that involve math, spatial awareness, and social reasoning. This helps train AI to understand images and instructions more accurately.
What this means in practice
- •For ai developers: Train multimodal AI models using curated image-instruction pairs to improve understanding of complex reasoning tasks.
- •For data annotation teams: Use the pipeline to generate high-quality instruction-response pairs for image datasets to efficiently expand training data.
Authors
Eleni E. Kasoura, Dimitrios Damianos, Georgios Paraskevopoulos, Vassilios Katsouros
Abstract
This work presents a compact and reproducible pipeline for synthesizing and curating image-based instruction-response pairs for DCVLR. We draw from the Cauldron collection and use Magpie-style instruction and response synthesis with Qwen2.5-VL-7B Instruct to create concise, image-conditioned instruction pairs. The pipeline consists of instruction synthesis, conditioned on input images and controlled via a system prompt, sanitization, tagging, and filtering of the instructions based on the score obtained by a reward model (DeBERTa-v3). We retain samples with positive scores to maximize the quality of the generated instructions. We observe that score distributions have a median and mean reward score of ~ 1.53 and ~ 1.75 respectively, indicating that most generated instructions are of good quality. The resulting dataset contains diverse instructions in the math, spatial, and social reasoning domains.