Extended reality offers new ways to study human brain and behavior

Moving Beyond the Flat Screen: Opportunities and Challenges of Extended Reality for Studying Human Brain and Behavior

Human-Computer Interaction

Summary

Most brain and behavior studies happen with people sitting still looking at flat screens, which doesn't fully match how we act in real life. The authors discuss how extended reality (XR), which includes virtual reality and augmented reality, lets researchers create more lifelike experiments where people move and interact naturally. These XR studies could reveal more about attention, memory, and language, but there are still challenges like hardware limits and measuring brain activity accurately. The authors highlight both the exciting possibilities and the practical problems of using XR in brain research.

What this means in practice

  • For neurotechnology developers: Create immersive cognitive testing tools that allow active, natural engagement to measure brain function beyond flat screen limitations.$Commercial implications: This paper supports development of XR-based brain assessment products that provide more realistic cognitive testing environments than current tools.
  • For human factors engineers: Design XR experimental setups that enable realistic multitasking and natural responses to study behavior under controlled but lifelike conditions.

A position paper. It proposes an approach and reports no results.

Authors

Tom Bullock, Radha Kumaran, You-Jin Kim, Emily Machniak, Tobias Höllerer, Barry Giesbrecht

Abstract

Much of what we know about human perception and cognition comes from austere experimental settings: static stimuli presented to sedentary observers on flat monitors, complex responses reduced to simple button presses, and brain activity recorded from immobilized heads using EEG or fMRI. These highly controlled paradigms have provided remarkable insights, but capture a small and often unusual slice of behavior, raising questions about how well the findings generalize to the real world. Extended reality (XR), spanning both immersive virtual reality (VR) and augmented reality (AR) that overlays digital content on the real world, offers a way beyond these constraints, supporting controlled, reproducible experiments in which participants actively engage with three-dimensional tasks. Drawing on a growing body of VR and AR studies of attention, memory, and language -- our own and others', some involving EEG -- we outline what XR makes newly possible for cognitive neuroscience and reflect on the practical and conceptual obstacles that remain. The opportunities are substantial, but realizing them requires solving practical problems of hardware, timing, and measurement, and being realistic about what a virtual or augmented world can and cannot stand in for.
View PDFOpen arXiv