VocalEyes improves speaker tracking in augmented reality captions
VocalEyes: Speaker-Aware Augmented Reality Captioning through In-Conversation Registration
Human-Computer Interaction
Summary
People using augmented reality captions to read speech can have trouble knowing who said what, especially in new groups. The authors developed VocalEyes, a system that remembers who is speaking by using natural self-introductions instead of needing special registration ahead of time. In a study, VocalEyes showed it could identify speakers with good accuracy and helped users better follow who was talking, while making the task easier. This system connects speech, speaker names, and faces visually to improve conversations and meeting reviews.
What this means in practice
- •For video conferencing service providers: Add speaker-aware AR captions using natural introductions to improve remote meeting clarity and post-meeting review.$Commercial implications: Enables new captioning features that link speech to speaker identities for meeting platforms enhancing user experience.
- •For assistive technology developers: Build AR caption interfaces that help hearing-impaired users track conversations by showing who is speaking in real time.
Authors
Yuxiao Wang, Xulong Tang, Chen Chen, Rawan alghofaili
Abstract
Co-located augmented reality (AR) captions make speech readable, but they can separate an utterance from the person who produced it. In unfamiliar groups, losing that source complicates immediate responses and later review: users must recover not only what was said, but also who said it. Conventional diarization returns anonymous clusters, while speaker recognition typically assumes pre-meeting enrollment. We built VocalEyes, a speaker-aware AR captioning system that creates named voice profiles from natural self-introductions. The interface coordinates speaker-attributed captions, a fixed profile card, and a visual cue that marks the articulating face. In a controlled within-subjects study with 20 participants who reported typical hearing, VocalEyes identified speakers with 88.0% accuracy and increased participant speaker-tracking accuracy from 47.2% with caption-only AR to 87.3% with the complete interface. Participants also reported lower workload with the complete interface. These findings show how in-conversation registration can preserve speaker attribution across live captions and meeting records in scripted small-group meetings.