SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new dataset called SI-Data to help computers better understand how to edit images based on both drawings (sketches) and user instructions. They used advanced language models to automatically generate examples showing the original image, the sketch, the instructions, and the edited image. Using this dataset, they built a system named SI-Edit that carefully combines the shape from the sketch with the meaning from the instructions to make precise changes to images. They also made new ways to check how well the edited images match both the sketches and instructions. Their experiments show that SI-Edit can make more accurate and detailed edits than previous methods.
Generative modelsSketch-based image editingGeometric constraintsSemantic instructionsMultimodal large language modelsDataset synthesisStructural fidelitySemantic adherencePixel-level refinementEvaluation metrics
Authors
Weixin Ye, Wei Wang, Hongguang Zhu, Xuecheng Nie
Abstract
Despite rapid advances in generative models, achieving pixel-level precision in sketch-based image editing remains a persistent challenge, particularly for fine-grained local deformations. This gap stems primarily from the critical shortage of high-quality, publicly available benchmark datasets that jointly provide geometric constraints and semantic instructions. To address this issue, we first introduce **SI-Data**, a high-quality dataset specifically designed for instruction-guided local sketch editing. We develop an automated pipeline leveraging Multimodal Large Language Models (MLLMs) to synthesize comprehensive quadruplets comprising original images, local geometric sketches, semantic instructions, and corresponding edited images. By providing both reliable spatial anchors and explicit semantic intent, SI-Data uniquely enables collaborative spatial-semantic learning. Building upon this, we propose a collaborative framework called **SI-Edit** that integrates semantic instructions with precise geometric constraints. Furthermore, to address the lack of standardized evaluation, we establish a comprehensive set of metrics designed to measure both structural fidelity (e.g., sketch-to-edge alignment) and semantic adherence. Experimental results demonstrate that SI-Edit provides more reliable structural control than baselines for sketch-based image editing, and achieves precise, pixel-level local refinements aligned with user intent. The data and code are released on the [project page](https://github.com/ywxsuperstar/SIEdit).