Vision language models improve self-training with program-verified checking
Program-Verified Self-Evolution for Vision-Language Models
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageMachine LearningNeural and Evolutionary Computing
Summary
Vision-language models learn by asking questions about pictures without needing human-provided answers. The problem is that guessing the correct answers can be wrong a lot of the time. The authors propose a new way where the model breaks down images into structured data like graphs and uses fixed programs to create questions and verify answers more accurately. This method leads to much more reliable answers and better overall model performance.
What this means in practice
- •For machine learning engineers: Improve training pipelines for vision-language models by integrating program-verified QA to boost answer accuracy without requiring human labels.
- •For software developers in ai tools: Create applications that generate and verify image-based questions automatically using structured representations for higher precision in QA systems.
Authors
Ahmed Heakl, Sungik Choi, Moontae Lee, Salman Khan
Abstract
Self-evolving vision-language models train on questions they generate from unlabeled images. Since these questions have no gold answers, prior methods label them by majority vote over sampled answers or by a model judge. In a human evaluation, we find that 24\% of majority-vote labels and 18\% of model-judge labels produced during self-evolution are wrong. To address this problem, we present Verifiable QA Generation for Self-Evolving Models (VQS), which changes how the model judges answers. Instead of voting on an answer, the model parses each image into a structured record, such as a scene graph, a chart table, or a diagram graph. Fixed programs then write a question from the record and compute its answer. The model still acts as a visual checker, but it only confirms the individual facts the program reads, one short claim at a time. These claim-level checks select the parser's training targets, so the parser also improves without labels. Human raters find 94\% of VQS answers correct, against 76\% for majority voting. Across ten benchmarks, VQS improves Qwen3-VL by up to 3.18 points at the 2B, 4B, and 8B scales and outperforms the strongest self-evolving baseline at each. Gains keep growing over three training rounds, reaching 3.84 points at 2B. Code is released at https://github.com/ahmedheakl/VQS