Reframe reduces gpu load and speeds up live speech models

ReFrame: Silence-Aware Compute Reclamation for Full-Duplex Speech Models

Machine Learning

Summary

Full-duplex speech models process every audio frame continuously during a conversation, which uses a lot of computing power. The authors found that by detecting when both people are silent or when one speaks and the other listens, they can skip or batch some calculations without losing accuracy. Their system, ReFrame, smartly reallocates computing time saved this way to improve other tasks like retrieving information. This approach cuts GPU time by up to 31% while keeping almost all text outputs correct and also helps increase correct spoken answers.

What this means in practice

  • For ai voice assistant developers: Reduce GPU usage during live conversations by skipping redundant computations when silence is detected, improving efficiency without lowering transcript quality.
  • For real-time speech recognition teams: Improve responsiveness and accuracy by reallocating saved processing time toward external retrieval and context-enhancement for voice agents during user interaction.
  • For customer support platform engineers: Use silence-aware inference to lower operational costs of full-duplex speech models in live dialogue services, enabling better scalability.$Commercial implications: This system enables more efficient live speech-based customer support services, reducing server costs and improving interaction quality.

Authors

Zheng Zhou, Guanyi Chen, Jing Wei, Haoyuan Liang, Jian Zhang, Haohuan Fu

Abstract

Full-duplex speech models execute dense model inference for every audio frame throughout a live interaction. This uniform framewise schedule creates sustained GPU load across dialogue states whose computation has strong, reusable temporal structure. ReFrame formulates full-duplex inference as a silence-aware computation problem. Mutual Silence Speculation (MSS) operates when both participants are silent: it evaluates multi-frame Temporal continuations, verifies text and hidden-state stability, and applies batched Mimi audio-token correction. Listening Frame Folding (LFF) operates while the user speaks and the agent listens: it processes already-arrived user frames through parallel Temporal execution and frame-batched audio-token generation while accepting only the verified prefix. Frame-Budget Reinvestment (FBR) assigns the released frame intervals to resumable external retrieval and language-model workers, then injects timely references through Moshi's conditioning interface. We instantiate ReFrame on five full-duplex models spanning Moshi-family, phase-structured, and flattened autoregressive generation. Across the complete 1,225-conversation FDBench, ReFrame reduces measured GPU execution time by 18.80-31.01% and retains 96.92-100% non-PAD text agreement on five different models. On HaluEval tests, FBR increases the number of correct generated spoken answers from 86 to 205 under the same auxiliary-service budget. ReFrame turns recurring dialogue structure into lower inference cost and higher end-to-end service capability.