Software engineering benchmark tests visual and textual code repair skills
SWE-PolyVision: Benchmarking Cross-Image Abductive Reasoning for Repository-Level Software Engineering
Software Engineering
Summary
Software bugs can sometimes be fixed by looking at pictures or videos related to the code, but it's hard for AI to combine all that visual information with text to find the right fix. The authors created a set of real-world tasks from open-source projects that test if AI systems can understand multiple images and videos alongside code. They found that having access to these visuals changes which tasks AI can solve, but it's not consistent yet. Their work helps show where AI still struggles with using pictures and text together to repair software accurately.
What this means in practice
- •For software engineering teams: Evaluate and improve AI tools that fix code using both code text and related images from software repositories.
- •For ai developers for coding tools: Test model improvements in understanding multiple visuals to enhance code repair assistance capabilities.
Authors
Jiajun Wu, Leixin Sun, Zihan Tan, Yitao Liu, Shuo Li, Jiaru Qian, Shanghaoran Quan, Chuangxin Zhao, Yangxu Liao, Yang Liu, Bin Chong, Guancheng Wan
Abstract
Current multimodal software-engineering benchmarks expose images as additional context, but do not test whether an agent can integrate evidence distributed across images into a verified repository-level repair. We present SWE-PolyVision, an executable benchmark of 92 real tasks from 36 open-source organizations, with 48 public tasks and 44 private holdouts. The release contains 402 static images and 6 videos, with at least two visual inputs per task. Each task pairs a fixed pre-fix repository with an isolated verifier and is evaluated under the supported conditions among three access modes: Text-only, Native Vision, and Tool-mediated Vision. Across eleven coding models, visual access changes which tasks are solved, but effects depend on both model and task. Two trace-linked Native Vision cases illustrate how complementary visual and textual clues can lead to source-localized, verified repairs; controlled interventions show that this conversion is not yet stable across inputs. SWE-PolyVision thus separates the availability of multi-image evidence from its successful use in repository-level repair, without treating patch success alone as proof of explicit reasoning.