Unified multimodal models improve themselves by swapping training tasks

Recursive Self-Improvement in Unified Multimodal Models

Computation and LanguageComputer Vision and Pattern Recognition

Summary

Unified multimodal models can understand and create both text and images. The authors propose a method where the model improves itself by generating images and then checking how well it understands those images using its own text skills. By writing simple programs to fix mistakes and verifying their correctness externally, the model avoids accumulating errors over time. This process helps the model get better at both creating images and understanding them, especially for chart-related tasks.

What this means in practice

  • For software development teams: Enhance AI-assisted tools that generate and verify visual content automatically to improve reliability without human supervision.
  • For data visualization engineers: Build AI systems that better understand and produce accurate chart visualizations by iteratively refining their own weaknesses.

Authors

Huijuan Wang, Chufan Shi, Cheng Yang, Yaokang Wu, Taylor Berg-Kirkpatrick, Xuezhe Ma

Abstract

Unified multimodal models (UMMs) understand and generate both text and images, which lets a model produce its own training data. Existing self-improvement in UMMs keeps supervision on the visual side, where image understanding judges image generation. We propose recursive cross-capability self-improvement (RSI), a training loop in which the text and visual abilities of a UMM supply training data for one another. In each round, the model generates images and reads them to find where it falls short. It then writes programs aimed at these shortcomings, and execution verifies every result against its specification. Verified renders train image generation, while labeled renders and the model's own correct programs train visual understanding and program writing. Program execution thus acts as a source of truth outside the model, so errors do not accumulate across rounds. We study RSI on charts and build BasicChartBench to evaluate open models early in training. On requests worded differently from training, four rounds of RSI raise the score from 45.7% to 60.2%, while continued training stays at 46.3%. Verified construction carries most of the gain, and targeting the model's failures adds 3.5%. Along the way, the share of verified programs rises from 48.9% to 95.2%, and the reader's accuracy on edited renders rises from 55.6% to 87.4%.