Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR
2026-07-06 • Sound
Sound
AI summaryⓘ
The authors propose LatentASR, a method that improves speech recognition by adding a small, trainable component to an existing frozen model without changing it directly. This component refines some internal settings step-by-step and decides if more effort is needed to understand difficult spoken inputs. They show that unlike other methods that need lots of data, LatentASR helps reduce errors especially on hard and accented speech, even with very little training data. Their method also saves computational resources by stopping early when extra work isn't helpful.
ASRend-to-end modelstest-time adaptationlatent vectorsWERfrozen backboneparameter-efficient tuningaccented speechdynamic haltingcode-switching
Authors
Ho Lam Chung, Yiming Chen, Dau-Cheng Lyu, Hsiao-Tsung Hung, Hung-yi Lee
Abstract
End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous latent test-time scaling to a frozen ASR backbone. Two small trainable modules drive it: a Latent Adapter that iteratively refines a few latent prefix positions through bounded, stabilized updates, and a Value Head that predicts whether extra computation will help and halts the loop early. The Qwen3-ASR-0.6B backbone stays fully frozen, and we train only ~4M extra parameters. We activate this loop with a deliberately small, diverse 500-utterance training set. Under this minimal-data regime, standard adaptation methods all regress: full fine-tuning, LoRA, and prompt tuning each increase WER. LatentASR is the only tested method that reduces WER on both clean benchmarks (FLEURS -2.54% and VoxPopuli -0.47% relative). The reductions are concentrated on intrinsically hard inputs. On accented and code-switched speech (ASCEND), LatentASR achieves a 16.0% relative CER reduction. Across 30 FLEURS languages (23,049 utterances), the multilingual WER decreases uniformly across resource tiers, confirming that the adapter generalizes without overfitting. Dynamic halting preserves most of the clean-set reduction at a fraction of the compute, skipping roughly half of all utterances at the entry gate. Our results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.