HiPLEX improves real time speech model timing and content coordination
HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models
Sound
Summary
Having conversations with AI is tricky because the model needs to know both what to say and when to say it. The authors created HiPLEX, a new way to help AI models decide the timing and content of spoken replies separately but in sync. This method helps the AI better handle interruptions, pauses, and back-and-forth chatter, making conversations feel more natural. Tests showed HiPLEX reduces awkward interruptions and matches human timing better than previous approaches.
What this means in practice
- •For voice assistant developers: Improve real-time conversational AI by coordinating when and what speech is emitted for smoother user interaction.
- •For call center technology teams: Enhance automated dialogue agents to manage turn-taking and interruptions more naturally during customer calls.
Authors
Kyudan Jung, Hyunsin Park, Yoonhyung Lee, Jinhwan Park, Jinhyeok Yang, KiHyun Nam, Jaegul Choo, Jinkyu Lee
Abstract
As human--AI interactions become more conversational, full-duplex speech language models capable of natural real-time dialogue are growing in importance. Beyond generating appropriate responses, these models must coordinate turn-taking, backchanneling, and floor management in real time. Reinforcement learning (RL) provides a way to refine these behaviors through direct feedback on interaction outcomes. However, existing RL methods either apply timing feedback to a token policy or optimize semantic content, leaving the joint improvement of timing and content unresolved. We introduce HiPLEX, an RL framework that factorizes a pretrained full-duplex text policy into a control policy that decides when to emit content and a conditional content policy that decides what to emit. The first factor selects among 'pad', 'epad', and 'con'. The second selects a token only when 'con' is chosen. This hierarchy describes conditional actions within each frame and uses the model's existing text head. We route timing advantages to the token-group factor through event-causal masks derived from generated speech episodes, and route an LLM-judge semantic advantage to the conditional content factor. Across three Moshi seeds on Full-Duplex-Bench v1, HiPLEX reduces takeover rates during natural user pauses and backchannel opportunities, and shortens post-interruption response latency relative to GRPO, while maintaining comparable judged interruption-response quality. On Moshi and PersonaPlex, HiPLEX better matches pooled human turn-timing and backchannel-rate marginals than GRPO.