Papers for

automotive user interface designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multimodal system improves identifying landmarks from vehicle passengers

Speak to the City: Multimodal Resolution for Outside-the-Vehicle References

Abstract: As autonomous vehicles and Extended Reality (XR) headsets enable novel in-car interactions, seamlessly querying physical landmarks, known as Outside-the-Vehicle Referencing (OVR), remains challenging due to ego-motion and referential ambiguity. We present a robust, multimodal OVR framework fusing user gaze and natural language to identify Points of Interest (POIs). To address the scarcity of dynamic vehicular data, we developed a VR-based pipeline synchronizing 360-degree transit videos with vehicle GNSS telemetry. Through a user study (N=46) mapping passenger head orientation into a 3D geospatial Digital Twin, we captured authentic gaze-speech behaviors. We subsequently trained a lightweight Transformer network, leveraging LLMs to dynamically align continuous spatial gaze vectors with discrete verbal context. Experimental results demonstrate high accuracy and low computational overhead, achieving an 83.33% Top-1 accuracy (87.72% Top-2) and an average inference time of 24.3 milliseconds. This real-time paradigm effectively resolves referential ambiguity, enabling context-aware spatial retrieval for passengers within the vehicle.

Sun 13 SeptHuman-Computer InteractionComputation and LanguageInformation Retrieval
The gist
It is hard for passengers in moving cars to point out landmarks using just words because things outside keep changing and words can be unclear. The authors created a system that combines where a person is looking with what they say to better figure out which landmark they mean. They gathered data using virtual reality and maps to train a smart model that understands both speech and gaze in real time. This method works quickly and accurately to help passengers communicate more clearly about locations outside the vehicle.
Open → 2609.14691v1

Communication timing improves in-car agent interactions and reduces interruptions

When2Talk: When Should a Proactive In-Car Agent Talk?

Abstract: Proactive in-cabin agents can help passengers understand automated-vehicle (AV) behavior, but communicating every ride event may introduce unnecessary interruptions. We investigated how communication should adapt to event priority and passenger activity. In a mixed-methods within-subject study, 41 participants rode as passenger in a VR simulated fully-automated vehicle. We compared an event-triggered (ET) policy that communicated immediately at every event with a context-sensitive (CS) policy that selected \textit{Immediate}, \textit{Delayed}, or \textit{Silent} communications. CS increased communication appropriateness and substantially reduced perceived interruption. Perceived trust did not differ between policies, although baselines dispositional trust differentiated communication preferences. Findings highlight event consequence, passenger activity, continuing information value, and confirmation need as key considerations for selective in-cabin communication.

Fri 11 SeptHuman-Computer InteractionComputer Vision and Pattern Recognition
The gist
Automated cars can use in-car agents that talk to passengers to explain what the car is doing. The authors found that if these agents speak only when important and consider what the passenger is doing, people feel less interrupted and find the communication more appropriate. They tested two ways: one that talks about every event immediately, and another that chooses when to talk or stay silent based on the situation. The results showed that selective talking leads to better experiences without reducing trust in the car system.
Open → 2609.12503v1