Telugu spoken question answering benchmark reveals language and speech challenges
VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering
Computation and Language
Summary
Answering questions spoken in Telugu with computers is hard because the language is less studied, and speech can be tricky to understand automatically. The authors created VākQA, a dataset of Telugu questions and answers with audio, text, and verified answers to help test systems. They checked how well automatic scoring methods work compared to humans and found some methods are too strict or unfair to different ways of saying the same answer. Using this, they tested various models and showed that Telugu’s unique expressions and errors from speech recognition and translation can make answering questions difficult.
What this means in practice
- •For voice assistant developers: Improve Telugu-language question answering by testing systems with VākQA’s authentic speech and text data covering multiple domains.
- •For speech recognition engineers: Analyze how phonetic confusions and cascaded errors impact Telugu spoken question answering to better tune ASR and MT components.
Authors
Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D, Srihari Bandarupalli, Santosh Kesiraju, Anil Vuppala
Abstract
Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. We first validate evaluation methods against human judgements: Gemini-as-a-judge best approximates human ratings but is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference. Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively. VākQA is publicly released.