Voice controlled tool simplifies XR experiment creation for transport research
XARA: A Multimodal Agentic System for eXtended Reality Experiment Generation
Human-Computer Interaction
Summary
Creating virtual reality experiments to study transportation is usually hard and requires special technical skills. The authors developed XARA, a tool that lets researchers use their voice and hand gestures to build these virtual scenes quickly in a 3D space. This system translates spoken commands into actions that change the virtual world in real time, making it easier to test things like driving and pedestrian behavior. By simplifying the process, XARA helps experts design and change their experiments faster without needing deep knowledge of game engines.
What this means in practice
- •For transportation planners: Design interactive virtual scenarios using voice commands to test urban and traffic designs for behavioral impact.
- •For human factors teams: Rapidly prototype and modify immersive safety and travel behavior experiments without coding expertise.
Authors
Tareq Alsaleh, Thomas Zhao, Luke Guardino, Keagan Valentine, Isabel Kim, Bilal Farooq
Abstract
Controlled experiments are essential for advancing transportation research, yet designing eXtended Reality (XR) environments for behavioural studies remains technically demanding and inaccessible to many domain researchers. Current workflows require proficiency in game engines such as Unity or Unreal Engine, imposing significant barriers to rapid scenario prototyping and iterative experiment design. This paper presents XARA, a novel system that integrates a dual large language model (LLM) architecture with an immersive XR environment to enable voice driven experiment generation for transportation research. XARA employs a pipeline in which spoken natural language commands are first transcribed and then interpreted into structured commands that manipulate a Unity based XR scene in real time. The system supports object creation, spatial manipulation, animation control, and scene persistence through a modular architecture. By lowering the technical threshold for XR experiment creation, XARA enables transportation researchers to rapidly prototype driving, pedestrian, and urban design scenarios using intuitive voice commands and hand gestures within an immersive 3D environment. This paper describes the system architecture, the voice to command pipeline, and discusses potential applications across behavioural travel experiments, safety studies, human factors research, and policy testing.