Scenario knowledge guides when and how to help kiosk users
A Scenario-Knowledge-Driven Pipeline for Just-in-Time Assistance
Human-Computer Interaction
Summary
Many kiosks don’t know when a user is struggling, and helping at the wrong time or in the wrong way can be unhelpful. The authors created a system that uses a single document with detailed knowledge about the scenario to decide when and how to assist someone quietly struggling on a kiosk. Their system separates narration, assessment of need, and help proposals, so each can be checked independently. Testing on recorded kiosk sessions showed it could accurately detect struggles, decide when help is needed, and avoid unnecessary interruptions.
What this means in practice
- •For self-service kiosk developers: Provide precise, verifiable assistance prompts that activate only when users show real signs of struggle based on a human-written scenario document.$Commercial implications: Enables kiosk system makers to sell smarter help features that improve user experience and reduce unnecessary interruptions.
- •For customer support operations: Use scenario-knowledge-driven assessments to monitor and escalate customer difficulties accurately during unattended service sessions.
Authors
Zhiyuan Li, Tatsunori Hara, Jun Ota
Abstract
Detecting a silently struggling kiosk user is only the first step; deciding whether, when, and how to help depends on scenario knowledge usually buried in model weights and thresholds. We propose a scenario-knowledge-driven pipeline: a single scenario knowledge document, human-authored and version-controlled, configures sensing, constrains LLM reasoning, and shapes a graded intervention proposal. Narration, assistance-need assessment, and proposal are kept separate for independent audit. As proof of concept, we replay two recorded kiosk sessions offline, chosen before the runs for their struggle evidence and retrospective detail. Both cases support what the design promises: checkable reporting and measured escalation. Across 95 updates, every sentence of the append-only narration cites the primitive events underlying it, and the rule layer detects 12 of 13 and 7 of 7 annotated struggle episodes under a strict criterion. The assessor de-escalates on recovery and reaches the top rung exactly once, under maximally converging evidence. At the decisive help-seeking turn, narration, assessment, and the participants' retrospective accounts converge. The appropriateness of these interventions, the pipeline's restraint on sessions without struggle, and the document's transfer to a new scenario frame the agenda.