VisualRepair: Dynamic Tool Calling and Region Focusing for Visual Software Issue Repair
2026-07-15 • Software Engineering
Software Engineering
AI summaryⓘ
The authors address the problem of fixing software bugs by using pictures like screenshots that show what went wrong. They created a system called VisualRepair that first figures out what kind of image it’s looking at and then focuses on the important parts of the picture to better understand the bug. This helps the system find more accurate fixes and work better than previous methods. Their tests show that VisualRepair performs better at fixing bugs in software reports with images.
Automated Program RepairLarge Language ModelsMultimodal LearningBug ScreenshotsImage ClassificationFault LocalizationVisual Software Issue RepairRegion FocusingSWE-bench Multimodal benchmark
Authors
Jingyu Xiao, Zhongyi Zhang, Haoran Hou, Yuxuan Wan, Yuan Jiang, Yintong Huo, Michael R. Lyu
Abstract
Automated Program Repair (APR) has witnessed significant progress with the advent of Large Language Models (LLMs). However, as modern software systems increasingly expose rich graphical user interfaces, effectively leveraging visual information from bug screenshots has become essential for understanding bugs and generating accurate fixes in multimodal scenarios. Real-world issue reports frequently contain heterogeneous visual attachments including UI screenshots, IDE snapshots, GIFs, and text-centric images, each with distinct visual patterns and domain-specific semantics that impose substantial perceptual demands on MLLMs. Furthermore, bug screenshots often contain large expanses of uninformative and bug-irrelevant regions, distracting the model's attention and limiting patch diversity. To address these challenges, we propose VisualRepair, an MLLM-based framework for visual software issue repair comprising two core modules: Image Type-aware Tool Calling (ITTC), which classifies input images and dynamically invokes a tailored tool-calling chain for robust visual interpretation, and Dynamic Test-time Region Focusing (DTRF), which grounds multiple bug-related region candidates and refines them via an adaptive zoom-in and zoom-out strategy to improve fault localization and promote diverse patch generation. Extensive experiments on the SWE-bench Multimodal benchmark demonstrate that VisualRepair consistently outperforms state-of-the-art approaches. VisualRepair resolves 196 and 25 instances on the test and dev sets, respectively, surpassing the best baseline by 10 and 11 instances. These results highlight the effectiveness of type-aware visual understanding and region-focused localization for automated visual software issue repair.