Not All Fallbacks Are Failures: Understanding and Recovering from Fallbacks in Mobile Voice Assistants
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how a smartwatch voice assistant handles situations when it misunderstands or can’t process what a user says, caused by noise or unclear speech. They collected data from over 500 users and categorized different types of fallback errors. They then tested various computer models to see which best identifies these errors while using less computing power. Their findings show that simpler models using embeddings work better than bigger, more complex ones for this task. The authors also share practical advice for improving voice assistant responses in real-world settings.
voice assistantsfallback handlingspeech recognition errorsembedding classifiersgenerative modelsnatural language processingsmartwatchuser interactioncomputational efficiencytaxonomies
Authors
Phillip Schneider, Alexandre Mercier, Joshua Oehms, Kristiina Jokinen, Florian Matthes
Abstract
Robust understanding of user input is a core requirement for voice assistants deployed in real-world environments. In practice, these systems encounter heterogeneous fallback situations caused by noisy audio input, transcription errors, ambiguous requests, incomplete utterances, or unintended activations. Existing systems typically respond with generic fallback messages, which do not resolve the underlying interaction failure and can degrade user experience. We study fallback handling in a deployed smartwatch-based voice assistant for general health support in everyday environments. Our analysis is based on six months of real-world usage data from more than 500 users, yielding a dataset of 3,030 anonymized, naturally occurring fallback-triggering utterances. We contribute (1) an operational taxonomy and the annotated VoxFallbacks dataset of these interactions, (2) a comparative evaluation of different models within a classification pipeline under practical deployment constraints, and (3) practical lessons for designing robust and cost-efficient fallback mechanisms. Results show that lightweight embedding-based classifiers outperform larger generative models on most classification tasks while requiring substantially fewer computational resources.