Ai uncertainty improves multisensory visualization interactions
When Direct Manipulation Becomes a Guess: Productive Friction in AI-Mediated Multisensory Visualization
Human-Computer Interaction
Summary
When people use tools that combine AI and senses like speech or touch, it can feel like the tool always works perfectly. The paper explains that this feeling hides how uncertain AI predictions really are, which can cause confusion. The authors propose making this uncertainty clear through sensory feedback, like changing sounds or visuals, so users understand the AI's limits. This helps people work more effectively with AI by recognizing when the system is guessing and when it is confident.
What this means in practice
- •For user interface designers: Create interfaces that show AI uncertainty with touch, sound, or visuals to improve understanding and control in AI-assisted tools.
- •For virtual reality developers: Incorporate sensory cues that reflect AI's guessing into VR systems to make user interactions with uncertain AI outputs clearer and more manageable.
A position paper. It proposes an approach and reports no results.
Authors
Anchit Mishra
Abstract
Generative visualization increasingly embeds large language models within direct manipulation and multisensory interaction. Speech, gaze, touch, gesture, sound, and haptics can make probabilistic inference feel like familiar, deterministic tool use. I call this gap a deterministic-affordance mismatch: deterministic interaction cues persist while AI weakens predictability, locality, reversibility, or provenance. Reading the malleable interfaces of Iron Man 2 against systems from Blade Runner 2049, I propose productive friction, including cross-sensory renderings whose detail reflects model uncertainty. Four frictions expose the inference boundary, match action to effect, make stochastic branches tangible, and attribute sensory agreement.