CanvasObs improves object editing accuracy on infinite 2D canvases
CanvasObs: Structured Spatial Observation for Agent Editing on 2D Infinite Canvases
Human-Computer Interaction
Summary
Editing objects on very large or infinite digital canvases is hard because some objects might be hidden, offscreen, or overlapping, making it tough for computer agents to know exactly what to edit. The authors introduce CanvasObs, a new way to give the editing agent extra structured information that describes objects, their location, visibility, and relationships. In tests across many trial runs with different models, CanvasObs helped agents select the correct object much more often than before. This shows that adding structured understanding helps editing tools work better without needing to redesign the whole system.
What this means in practice
- •For user interface developers: Enhance digital canvas editing tools to accurately select and edit hidden or overlapping objects using structured spatial data.
- •For collaborative design teams: Improve remote teamwork on complex visual projects by enabling accurate offscreen object interaction and state tracking.
Authors
Jiahao Xu, Haohui Wang, Rebecca Faust, Dawei Zhou, Chris North
Abstract
Editing agents on 2D infinite canvases must reason about application state that pixels may not reveal. Targets may be offscreen or covered, while visual containment and touching arrows may disagree with recorded frame membership or connector bindings. We ask how a structured observation layer can support accurate object selection during agentic editing on 2D infinite canvases. We present CanvasObs, which augments visual context with a structured payload and guidance for reading four components: Identity & State, Visibility & Structure, Relations, and Offscreen Object bounds in action coordinates. A Remote Team Handoff case study illustrates retrieving an editable offscreen copy, choosing a covered chart by color, separating cards by frame membership, and preserving an evidence connection. We turn these cases into ten controlled configurations and compare CanvasObs with the baseline agent in 1,200 browser trials across three models. CanvasObs improves Correct target for every model. Combined across models, it raises Correct target from 16.5% to 48.3%, an increase of 31.8 percentage points (95% confidence interval [29.0, 34.7]), and Full task success from 4.3% to 25.0%. These results show that compact structured observation can improve object choice on 2D infinite canvases without redesigning the editing pipeline.