Contextual pictograms help fix speech transcription errors in French and Spanish
Contextual Pictograms for Repairing Multilingual ASR Errors
Computation and LanguageHuman-Computer Interaction
Summary
Sometimes, when computers transcribe speech, they make mistakes that are hard to spot because the wrong words look or sound alike. The authors created an interface that not only shows possible word alternatives but also uses pictures related to uncertain words to help users understand what was really said. They plan to test this system with French and Spanish phrases to see if images help people better guess the speaker’s intended meaning compared to text alone. They also want to measure whether confusing pictures slow down the process or make misunderstandings worse.
What this means in practice
- •For call center operators: Use pictogram-enhanced transcripts to better interpret ambiguous customer speech in French and Spanish and improve response accuracy.
- •For multilingual voice assistant developers: Integrate contextual pictograms when speech recognition is uncertain to help users clarify voice commands in French and Spanish interfaces.$Commercial implications: Enables creation of voice assistants that reduce misunderstanding via visual support, improving user satisfaction and differentiating product features.
Authors
Mariana Hernandez, Hector Luna, Carlos Bruno, Luis Ricardo Santos, Elena Simoes
Abstract
Speech transcription errors can be difficult to recognize when competing word hypotheses are unfamiliar or visually similar. We describe an interface that combines alternative transcripts, replay and contextually checked pictograms for selected uncertain concepts. The system will initially support French and Spanish functional utterances. A counterbalanced study will assess whether pictorial feedback helps listeners recover the speaker's intended action beyond text alternatives alone. We will also quantify semantic mismatch and additional interaction time, since an inappropriate image can reinforce the wrong interpretation rather than repair it.