Spoken Wikipedia audio aligned with slide decks for better speech recognition
The Spoken Wikipedia Presentation Corpus
Computation and LanguageMultimediaSound
Summary
Automatic speech recognition (ASR) systems work better when they understand the context of what’s being said. The authors extended the Spoken Wikipedia Corpora by adding slide decks generated with the help of large language models (LLMs). These slides summarize different sections of spoken articles with titles, bullet points, visuals, and takeaway messages to give extra clues that can help ASR. They tested various speech and language models and found that error rates are lowest for English audio. The combined use of audio and slides shows promise for improving recognition by providing multimodal context.
What this means in practice
- •For speech recognition engineers: Use aligned Wikipedia audio and LLM-generated slides to improve recognition accuracy by providing multimodal context.
- •For multimodal ai developers: Develop zero-shot ASR models that incorporate visual slide information alongside audio for richer contextual understanding.
Authors
Thomas Ranzenberger, Steffen Freisinger, Tobias Bocklet, Korbinian Riedhammer
Abstract
We present the Spoken Wikipedia Presentation Corpus, an extension of the Spoken Wikipedia Corpora featuring LLM-generated slide decks for multimodal ASR. Slides are created from LLM-segmented sections using a hybrid pipeline that combines LLM-based content planning with rule-based design decisions. For each section, an LLM generates a slide title, bullet points, a takeaway message, and a visual description that is used to create an illustration. Rule-based matching then selects layouts, themes, and styles to produce the final slides. A vision LLM extracts slide text as Markdown. We evaluate multiple ASR and spoken language models (SLMs). The best model achieves an average micro-WER of 10.23% and an average micro-CER of 6.48% on audio-only inputs. English yields the lowest error rates, followed by German and Dutch, while performance declines across lower-resource languages. Although audio-only baselines are strong, multimodal zero-shot prompting of omni models remains challenging. The aligned slide, text, and audio data show a strong potential to improve recognition through cross-modal context.