Ar glasses offer adaptive LLM-powered guided tours for urban heritage sites

ECHOES: An Adaptive LLM-Powered AR Glasses Docent for Evidence-Grounded Urban Heritage Interpretation

Human-Computer Interaction

Summary

Guided tours of historical city sites usually follow fixed scripts, which don't easily answer visitor questions or adapt to their interests and languages. The authors created ECHOES, a system using augmented reality glasses and a large language model (LLM) that listens to visitors’ questions and adapts its answers accordingly. It keeps track of what visitors ask about to give better responses and remembers language preferences for smooth conversations. The system also provides evidence-based information drawn from a custom-built historical database. Their prototype tested in Shanghai shows how this approach can make tours more interactive and personalized.

What this means in practice

  • For tourism operators: Provide interactive AR tours that adapt responses based on visitor questions, interests, and language preferences using an LLM-powered docent.$Commercial implications: This paper enables AR hardware providers and tourism companies to sell personalized heritage tour experiences powered by adaptive language models.
  • For museum technology teams: Integrate adaptive, evidence-backed LLM agents with AR to offer multilingual, conversational explanations of exhibits tailored to visitor curiosity.

Tested on one dataset.

Authors

Bang Xiao, Baosheng Jin, Jinhan Niu, Xingchen Zhang, Hua Shen

Abstract

Urban heritage tours are often delivered through fixed scripts, leaving limited room for visitors' situated questions, language preferences, or requests for evidence. We present ECHOES, an adaptive LLM-powered AR glasses docent for evidence-grounded interpretation of historical buildings. Specifically, the agent adapts along three dimensions: it modifies response structure according to the type of visitor question (identification, architectural detail, evidence-seeking, or route inquiry); it accumulates session-level interest signals from repeated questions and expressed curiosity to shape later explanations; and it tracks language preference and retrieval context across turns to maintain conversational continuity. The core of ECHOES is an LLM agent initialized as a source-aware urban heritage docent that realizes this adaptation through dialogue and preference context maintained across the session. Around this agent, ECHOES integrates visual site grounding, a self-built historical knowledge base supporting retrieval-augmented generation, and concise multilingual AR output designed for glanceable use. Through a working prototype centered on Shanghai's East Nanjing Road and the Bund, we demonstrate how an adaptive LLM-powered AR docent can transform historical tours from scripted narration into situated, question-responsive, and source-aware conversations.
View PDFOpen arXiv