Projection based appearance editing improves object visuals with scene response guidance
SREdit: Scene Response Guided Appearance Editing for Projection Mapping
Computer Vision and Pattern RecognitionMultimedia
Summary
Changing how an object looks using projected images is tricky because projectors can only compensate for distortions up to a limit and can't create completely new appearances. The authors found that common AI image editors don’t take into account how a real scene affects light, leading to errors when projecting edited images onto objects. They developed a method called SREdit that measures how different parts of an object respond to projected light and guides the editing process accordingly. This helps create better-looking projected images that match the physical object more accurately and makes the edits look more realistic.
What this means in practice
- •For projection mapping designers: Improve projected visual effects on physical objects by guiding image edits with surface light response for better accuracy and realism.
- •For event production teams: Create visually consistent and geometrically aligned projections on complex surfaces for immersive event displays using response-aware editing.
Authors
Yuqiao Xia, Yuchen Deng, Ziheng Chang, Bingyao Huang
Abstract
Projection-based appearance editing changes selected visual attributes of a physical object through projection mapping (PM). To display a desired appearance, PM typically uses projector compensation, which computes a projector input that counteracts the geometric and photometric distortions of the scene. Compensation can only approximate this appearance within the available projection range, and it cannot revise the appearance itself. Pretrained LLM-based image editors provide a direct way to produce the desired appearance from a text prompt or reference image; projector compensation can then attempt to reproduce the edited result through projection. However, these editors do not have an accurate measure of the scene geometry or scene response. Therefore, their outputs may contain geometric misalignment, inconsistent shading, or changes outside the available projection range, which downstream projector compensation cannot correct. To address this problem, we introduce SREdit, an efficient response-guided framework for projection-based appearance editing. SREdit estimates a scene response map from the reconstructed scene that reflects how strongly projected illumination can affect different surface regions. SREdit uses the source appearance to constrain the early sampling process and preserve registration with the fixed physical object, while the scene response map adapts the amount of editing across surface regions. Experiments across two pretrained image editors show that SREdit improves projected appearance quality and reproduction fidelity. It adds less than one second per image with negligible VRAM overhead. Participants also consistently preferred the results produced with SREdit in our user study.