Benchmark measures precise image edits on structured graphics
Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets
Computer Vision and Pattern Recognition
Summary
Fine-grained image editing means changing exactly what you want in a picture without messing up the rest. The authors found that existing tests either focus on realistic images with unclear scoring or on simple synthetic images that don’t fully match real uses. They created VeriEdit-Bench, a new test collection using detailed edited vector images where changes are exactly known. This lets them measure how well image editors follow instructions in multiple ways, showing that even the best models still make noticeable mistakes.
What this means in practice
- •For graphics software developers: Test and improve image editing tools using a benchmark that measures exact changes on vector graphics with clear, repeatable scoring.
- •For web interface designers: Verify that automated visual edits precisely modify intended elements in web interface graphics without unintended side effects.
Authors
Muyao Wang, Chen Zhu, Shiqi Yang, DongHyun Hwan, Hideki Nakayama
Abstract
Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely while leaving everything else intact. However, existing benchmarks leave a critical gap between realism and verifiability: benchmarks built on realistic images typically rely on human or vision--language model judgments, while deterministic evaluation has largely focused on synthetic shape canvases, with application-oriented extensions primarily limited to charts. This makes it difficult to determine precisely how much of a requested edit was executed, where unintended changes occurred, and whether small differences between models reflect genuine editing capability or evaluator uncertainty. To bridge this gap, we present VeriEdit-Bench, a benchmark for fine-grained, instruction-faithful image editing across realistic structured assets with deterministic, four-axis evaluation. Its 1,740 cases are compiled from the source code of 153 Scalable Vector Graphics (SVG) graphics, charts, web interfaces, and presentation slides. Controlled source-code edits preserve the original visual context while yielding exact target images, pixel-level edit masks, and explicit edit specifications, enabling reproducible scoring along four axes: edit fidelity, preservation, localization, and magnitude. Evaluating eleven editors, we find that even the strongest model remains far from full credit; rankings for the same recoloring operation reverse between charts and SVG graphics; and outputs with similar pixel-accuracy profiles can still differ substantially in localization and change magnitude. This decomposition yields graded, verifiable feedback and exposes model-specific capability and failure profiles that holistic scores or evaluator-dependent judgments may obscure.