Multimodal system improves identifying landmarks from vehicle passengers
Speak to the City: Multimodal Resolution for Outside-the-Vehicle References
Human-Computer InteractionComputation and LanguageInformation RetrievalMachine LearningMultimedia
Summary
It is hard for passengers in moving cars to point out landmarks using just words because things outside keep changing and words can be unclear. The authors created a system that combines where a person is looking with what they say to better figure out which landmark they mean. They gathered data using virtual reality and maps to train a smart model that understands both speech and gaze in real time. This method works quickly and accurately to help passengers communicate more clearly about locations outside the vehicle.
What this means in practice
- •For automotive user interface designers: Create in-car systems that identify landmarks accurately using combined gaze and speech inputs from passengers.
- •For extended reality headset developers: Develop XR experiences that interpret user speech and gaze together to interact with outdoor landmarks from inside vehicles.
Authors
Alireza Parchami, Artin Saberpour, Robin Connor Schramm, Jürgen Steimle, Ulrich Schwanecke
Abstract
As autonomous vehicles and Extended Reality (XR) headsets enable novel in-car interactions, seamlessly querying physical landmarks, known as Outside-the-Vehicle Referencing (OVR), remains challenging due to ego-motion and referential ambiguity. We present a robust, multimodal OVR framework fusing user gaze and natural language to identify Points of Interest (POIs). To address the scarcity of dynamic vehicular data, we developed a VR-based pipeline synchronizing 360-degree transit videos with vehicle GNSS telemetry. Through a user study (N=46) mapping passenger head orientation into a 3D geospatial Digital Twin, we captured authentic gaze-speech behaviors. We subsequently trained a lightweight Transformer network, leveraging LLMs to dynamically align continuous spatial gaze vectors with discrete verbal context. Experimental results demonstrate high accuracy and low computational overhead, achieving an 83.33% Top-1 accuracy (87.72% Top-2) and an average inference time of 24.3 milliseconds. This real-time paradigm effectively resolves referential ambiguity, enabling context-aware spatial retrieval for passengers within the vehicle.