Scribble only image editing models struggle to capture user intent
ScribbleEdit: A Benchmark for Scribble-Only Image Editing
GraphicsComputer Vision and Pattern Recognition
Summary
Using scribbles to tell a computer how to edit a picture is simple for people but hard for current AI models. The authors created a new test to see how well models understand these scribbles and follow the user's wishes. They found that current AI models don’t do a good job interpreting scribbles. To help improve this, they built a new method that better understands what scribbles mean. This work helps make editing tools that users can control easily with simple marks.
What this means in practice
- •For software developers: Create image editing apps that allow users to make precise edits with just scribbles to express their intent.
- •For graphic design studios: Improve digital art tools to support faster, more intuitive alterations based solely on user scribbles.$Commercial implications: Enables development of enhanced creative editing software targeted at professional designers requiring quick input methods.
Authors
Jie Ren, Hao Kang, Kai Guo, Yiding Yang, Bo Liu, Liming Jiang, Qing Yan, Zichuan Liu, Yizhi Song, Yue Xing, Hui Liu, Xin Lu
Abstract
Scribble-based interaction provides a lightweight and intuitive way for users to specify image editing intents in interactive editing tools. However, current image editing models based on VLMs or LLMs struggle to understand and execute edits based solely on scribble inputs. To systematically study this problem, we construct a new benchmark, ScribbleEdit, that evaluates the ability of image editing models to perform image editing conditioned on scribbles. This task requires both a deep understanding of the intention of the scribble and an accurate interpretation of its spatial information. In ScribbleEdit, we design an automated data construction pipeline and introduce a dedicated evaluation protocol that explicitly measures intention alignment. Our analysis reveals that existing VLM/LLM-based editing models fail to accurately capture scribble intentions. To guide future progress on scribble-only image editing, we propose a simple yet effective soft-token baseline, which enhances the model's understanding of scribble semantics and outperforms standard image editing models on our benchmark. Our evaluation and baseline together provide a concrete foundation for assessing and improving the scribble-driven image editing.