System combines speech and gestures to program 3D object movements
"Just Like This": Manner Deixis at the Graphics Interface
Human-Computer InteractionGraphics
Summary
People often say things like "open it like this" while showing how to move objects. This paper presents a system that lets users use words together with hand movements to teach 3D objects how to behave and move. The system shows a preview of the motion and lets users correct or clarify their instructions. In a study, this combination worked much better than using speech alone, making it easier and more natural to create animated behaviors.
What this means in practice
- •For 3d animation artists: Interactively teach and refine object movements using speech combined with hand demonstrations during animation design.
- •For virtual reality developers: Create intuitive authoring tools that let users specify behaviors in VR environments through combined speech and gestures.
Authors
Hamza El Alaoui, Jeffrey P. Bigham, Jun Rekimoto
Abstract
People communicate how things should move by combining words with demonstrations: "open it like this." We present an interaction technique that brings this expressive resource to conversational 3D authoring. Building on "Put-That-There," our system combines speech, pointing, and spatiotemporal demonstrations to specify editable behavior. A hand movement supplies evidence for a mechanism's axis, pivot, range, and pace; an animated preview makes the interpretation inspectable. Users refine behavior through further words or demonstrations, and the system can request a demonstration to clarify intent. In a twelve-participant study comparing three input configurations, our system achieved 89% participant-declared completion versus 47% with speech alone, had the highest observed match rates on six categorical accuracy measures and tied on two, and was preferred by nine participants. This work makes demonstration part of an ongoing authoring conversation: behavior can be shown, inspected, and revised.