Voice identity tokens help text AI track speakers across sessions
Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs
Sound
Summary
Keeping track of who said what is important in voice assistants that talk to many users. The authors created Speaker Handles, a way to give text-based AI models a clue about who is speaking by turning voice information into special tokens. These tokens help the AI remember speakers even when conversations happen at different times. They tested this idea with a new benchmark where it’s hard to guess who said what just from text, and the tokens worked well. This shows that linking voice identity to text AI can improve understanding in multi-user voice systems.
What this means in practice
- •For voice assistant developers: Add persistent speaker identity to text-based voice assistants for better multi-user dialogue management.$Commercial implications: Enables building commercial voice assistants that track speaker identity over multiple sessions for personalized interactions.
- •For customer support platform teams: Improve tracking of individual speakers in multi-user support calls where transcripts alone don’t identify users.
Authors
Runqiu Xu, Zhisheng Zheng, David Harwath
Abstract
Multi-user voice agents must track who said what across dialogue sessions. Text LLMs are attractive backbones for such agents, but transcripts alone do not expose acoustic speaker identity, leaving the model without a persistent reference for linking information to speakers across sessions. We address this gap by introducing Speaker Handles, soft-token representations that expose acoustic speaker identity to a frozen text LLM for cross-session speaker-dependent reasoning. A three-stage curriculum trains a lightweight projector, with fewer than 0.1% of the backbone's parameters, to map speaker embeddings into these handles. Establishing whether the resulting handles truly support cross-session speaker-dependent reasoning is challenging with existing benchmarks because textual cues can partially reveal fact ownership. We therefore present SpeakerBind, a controlled shared-agent benchmark in which overlapping facts across users require correct cross-session speaker attribution. Speaker Handles achieve 97.40-98.36% accuracy on VoxCeleb1 and 70.40% on SpeakerBind, close to the 71.88% topline. These results show that the proposed Speaker Handles provide an efficient way to integrate acoustic speaker identity into frozen text LLMs for speaker-content reasoning.