Mismatched views in video see-through extended reality cause new risks

Through Human Eyes and Machine Eyes: Understanding View Mismatch in Video See-Through Extended Reality

Human-Computer InteractionCryptography and Security

Summary

Video see-through extended reality systems capture what users see by taking screenshots, but these screenshots don't always match what the user actually sees. The authors studied this mismatch on a popular VR headset and found that the visible area to humans is often smaller and shaped differently than the rectangular images the system records. This difference can cause problems like leaking private information or missing important visual details in AI tasks that analyze these images. The paper highlights that this mismatch is more than just a technical issue—it can affect security, privacy, and reliability in these systems.

What this means in practice

  • For xr system developers: Design XR systems that accurately represent the user’s visible field to improve AI analysis and reduce privacy risks from mismatched views.
  • For security engineers: Identify potential privacy and security vulnerabilities caused by view mismatch in video see-through XR to develop better protections.

Tested on one dataset.

Authors

Yanming Xiu

Abstract

Video see-through extended reality (VST XR) systems commonly use headset screenshots or captured frames as proxies for the user's first-person visual context. However, the system-captured view and the user's effective visible field do not necessarily coincide: a screenshot records a rectangular machine-readable frame, whereas the user's effective visible region can be more constrained and non-rectangular. This paper studies this human-system view mismatch in VST XR. We formalize the relationship between the system-captured region and the human-visible region by defining their co-visible, system-only, and human-only regions. \rev{We then conduct a pilot-level boundary measurement on Meta Quest 3, revealing a clear mismatch between the rectangular screenshot frame and the approximate human-visible boundary. Building on this model, we analyze how view mismatch can affect screenshot-based XR sensing and downstream vision-language model tasks. Through four representative case studies, we illustrate potential risks and failure modes including prompt injection, privacy leakage, human-invisible information bias, and missing human-visible information. Our results show that view mismatch is not only a geometric artifact, but can also introduce security, privacy, and reliability concerns for AI-integrated VST XR systems.
View PDFOpen arXiv