Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking
2026-05-25 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study how to improve image editing guided by simple scribbles and text, especially when handling multiple editing tasks at once. They find that models struggle more with understanding diverse instructions than with differences in image types. To fix this, they create a two-step training process using lots of synthetic data followed by some real images, combine single-task examples to simulate multi-task learning cheaply, and focus loss calculations on parts of images that change. These methods noticeably improve performance on a benchmark for scribble-guided editing, and the authors plan to share their data and model.
scribble-guided image editingmulti-task learninginstruction-level generalizationsynthetic dataimage-domain generalizationcurriculum learningloss functionVIBE benchmarkimage editing
Authors
Mingyi Xu, Jinpeng Lin, Min Zhou, Tiezheng Ge, Ming Zeng
Abstract
Scribble-guided image editing allows users to combine simple scribble annotations with text prompts to specify both where and how an image should be edited, enabling flexible interaction with precise spatial control. However, existing models still exhibit unstable performance under this paradigm, especially in multi-task scenarios. To improve performance, we conduct empirical studies using an open-source editing model and reveal an asymmetry in generalization: instruction-level generalization, including across editing tasks and from single-task to multi-task settings, is more challenging than image-domain generalization, such as from synthetic to real-world images or from mosaicked to regular images. This suggests that the primary bottleneck lies in insufficient learning for diverse editing instructions rather than in the image domain gap. Motivated by this insight, we propose three strategies: (a) a Coverage-then-Realism Curriculum, a two-stage pipeline that first builds large-scale synthetic, instruction-rich data for broad task supervision, then curates a small set of real-world data to refine generation realism; (b) Multi-Task Mosaicking, which constructs multi-task training samples by concatenating single-task examples at nearly zero cost while enabling the learned capability to generalize to non-mosaicked images; and (c) an Edit-Focused Loss, which leverages the changed regions between input and output images in synthetic data to focus training on edited regions, improving both learning efficiency and editing accuracy. With these strategies, we substantially improve both single-task and multi-task scribble-guided editing on the VIBE benchmark, achieving state-of-the-art results. We will publicly release our dataset and model.