Omni-jev predicts smart assistant actions from user speech and text

Omni-JEV: Text and Dual-Stream Speech Decisions

Machine Learning

Summary

When you talk to a voice assistant, it needs to decide whether to respond, interrupt, or keep listening, based on what you say and how the conversation has gone so far. The authors developed Omni-JEV, a system that looks at both the words and speech signals from you and the assistant to choose the best next action from five possibilities. They trained the system using both supervised learning and reinforcement learning to improve its decision-making. This helps make voice assistants better at handling back-and-forth talk without confusion.

What this means in practice

Authors

Shengzhou Kong

Abstract

A user utterance does not uniquely determine the next action in full-duplex dialogue: its consequences depend on the assistant's response state and interaction history. Speech activity, utterance completion, and interruption intent provide related evidence, whose implications for a response require an explicit action policy. We formulate this task at collected-segment endpoints and introduce Omni-JEV, which maps text or role-indexed user and assistant-reference speech to probabilities over five interaction actions. An aligned speech-to-speech understanding backbone encodes the acoustic roles separately and shares its candidate readout with text inputs. Bilingual weak supervision supports cross-entropy fine-tuning followed by reinforcement learning for probability-sensitive decisions (RLCD). We compare Gaussian-logit and categorical Choice RLCD from the same supervised initialization under matched training exposure, observing different trade-offs between classification and probability quality. Gaussian Omni-JEV achieves test accuracies of 84.08% for audio and 96.55% for text. This common action interface supports decisions about when a full-duplex voice assistant should respond, preserve ongoing speech, interrupt, or resume an interrupted response.