RASSG reduces stale tool execution during streaming speech
RASSG: Repair-Aware Semantic Survival Gating for Streaming Tool Execution
Artificial Intelligence
Summary
When smart assistants try to do things while you are still speaking, they sometimes start actions too early and then have to correct themselves. The authors created a method called RASSG that waits for confirmation before starting a tool and can stop it if the speech changes. This makes fewer mistakes where tools run with wrong information and improves the overall experience. Their tests show much fewer errors and reasonable delays when using RASSG.
What this means in practice
- •For voice assistant developers: Improve timing and cancellation of tool launches during live speech to reduce errors from user self-corrections.
- •For call center automation teams: Enhance automated tool control to better handle customer speech corrections in real-time interactions.
Authors
Daijin Hu, Xi Wang
Abstract
Streaming speech assistants can launch tools before users finish speaking, overlapping execution with incoming speech to reduce user waiting time. However, subsequent self-corrections can invalidate apparently stable arguments, causing stale execution whose irreversible effects persist even after the final call is corrected. We propose Repair-Aware Semantic Survival Gating (RASSG) to jointly control tool launch and cancellation during streaming speech. RASSG launches a tool when three consecutive speech recognition results arrive with identical nonempty arguments, then combines calibrated argument invalidity risk with execution costs to decide cancellation at the last feasible opportunity before effects become irreversible. Evaluation on ReCAST, FDB-v3, and Audio2Tool uses a controlled execution clock with fixed tool runtimes and cancellation parameters. On ReCAST's prospective-repair subset, RASSG reduces the stale execution rate from LocalAgreement-2's 50.00% to 3.15%, with a mean delay of 3.676s across successful trajectories. The stale reduction also holds against every early-execution baseline on both public speech datasets. Ablations with identical initial launches support active cancellation, showing the value of controlling ongoing calls before their effects become irreversible. Code and data are available at https://github.com/daijinhu/RASSG.