Full-duplex dialogue models evaluated in two-way conversations
Conversation Is a Two-Body Problem: Dyadic Evaluation of Full-Duplex Dialogue Models
Computation and Language
Summary
Most voice systems speak and listen one at a time, which isn’t how people really talk. The paper shows that evaluating chat models that can talk and listen simultaneously needs both conversation partners to be active and reacting, not just one side. The authors created a way for two AI models to talk to each other in roles with goals and then scored both to see how they manage turns and interruptions. They found that each model’s behavior changes the other’s, so you can’t fairly judge one without looking at both. Their method doesn’t rely on fixed prerecorded audio, making it more realistic for these full-duplex systems.
What this means in practice
- •For voice assistant developers: Test two-way interactions of AI assistants by having full-duplex models converse under assigned roles and score both sides for better evaluation.
- •For call center automation teams: Evaluate conversational AI that must handle interruptions and overlaps by simulating dynamic dialogues between two full-duplex models fulfilling real roles.
Authors
Sungnyun Kim, Sungwoo Cho, Jihwan Oh, Se-Young Yun
Abstract
Full-duplex spoken dialogue models listen and speak at the same time, enabling voice agents to have natural, low-latency interactions that turn-based systems cannot offer. However, they are commonly evaluated against single-sided interlocutors: pre-recorded audio that cannot react, or an automated examiner that reacts in real time but only administers a fixed sequence of tests and is never graded. These single-sided frameworks evaluate only half of a two-body problem, where turn-taking, overlap, and interruption are joint products of two coupled speakers. We propose DyaFDB, a framework that evaluates full-duplex models in a dyadic setup: two models converse directly under assigned roles with cooperative or conflicting goals, and both sides are scored offline with an external judge. DyaFDB probes how the two models behave toward each other, such as how they take turns or carry an assigned role under different interests. We instantiate four tasks as 140 scenarios and record 7,560 conversations, covering six self- and cross-play pairings. Throughout the experiments, we observe that how a model behaves continually reshapes its partner. We thus demonstrate that each model must be both the examiner and examinee of the other, and no single fixed interlocutor can play both parts. We will release the scenarios, role prompts, and recording protocols between two full-duplex models, without any pre-recorded audio.