Intelligent interaction techniques promise mixed modality user interfaces
Intelligent Interaction Techniques (IIxT) - Proposal
Human-Computer InteractionArtificial Intelligence
Summary
Current computer user interfaces mostly use fixed, simple building blocks like menus and scroll bars, while AI-based interfaces mainly use chat windows separately. The authors suggest making these building blocks smarter so people can combine different ways of interacting, like voice and touch, more smoothly in the same app. This change would need new research on smarter building blocks and the tools to create them. It could also raise important questions about security, privacy, and money.
What this means in practice
- •For mobile app developers: Build apps that allow users to combine voice, touch, and typing seamlessly within the same interface element.$Commercial implications: Enables development of advanced multi-modal apps that enhance user experience and attract customers seeking smarter interfaces.
- •For cybersecurity teams: Design security measures that address privacy and data risks emerging from intelligent mixed-modality interaction components.
A position paper. It proposes an approach and reports no results.
Authors
Brad A. Myers
Abstract
Interaction techniques (IxTs) are the low-level, reusable components out of which user interfaces are designed, including menus, scroll bars, text input fields, and also copy-paste, text-entry, and selecting objects. The IxTs for graphical user interfaces (GUIs) were well established in the 1980s, with relatively minor additions and tweaks for smartphones in the 2000s. Most of today's AI user interfaces involve a chat window, which is an excellent interaction for some tasks, but is generally considered separate from the GUI IxTs. I argue for making the IxTs themselves more intelligent, so users can freely mix modalities, even within the same interaction. This will require research into new IxTs, and also into the infrastructure that will enable these intelligent IxTs (IIxTs) to be built. There are also significant security, privacy and economic implications to this vision.