Papers for

real-time transcription services

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Streaming omni-modal models improve accuracy by managing audio visual contradictions

Omni-Streaming Thinking

Abstract: Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence. Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average. We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.

Mon 14 SeptMachine Learning
The gist
When computers try to understand videos and sounds together as they happen, they sometimes make mistakes by assuming something too early and sticking to it, even if later sounds contradict what they thought. The authors call this problem premature cross-modal commitment. They designed a new method called Omni-Streaming Thinking (OST) that carefully tracks what evidence has been seen, what might come next, and marks ideas as 'pending' until it's sure they are correct. This way, OST can change its mind if new evidence disagrees and only answers questions when confident. Their tests show OST works better than other models at understanding streaming video and audio.
Open 2609.15128v1