GazeAgent improves eye tracking accuracy on AI glasses without user interruption
GazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses
Human-Computer InteractionComputer Vision and Pattern Recognition
Summary
Eye-tracking in AI-powered glasses usually needs users to stop and manually calibrate for accuracy, which can be annoying and can lose accuracy if the glasses move. The authors created GazeAgent, a system that automatically improves eye-tracking by watching how people naturally look, move, and interact with their hands through sensors. It finds moments when users focus on important things and uses those moments as calibration points without interrupting the user. Their tests showed this method reduces errors and users preferred it over manual calibration.
What this means in practice
- •For wearable device developers: Build AI glasses that continuously self-correct gaze tracking accuracy during natural use without requiring manual calibration breaks.$Commercial implications: Enables production of user-friendly AI glasses with adaptive gaze accuracy, appealing to consumer and enterprise markets.
- •For human-computer interaction engineers: Incorporate natural gaze calibration into multimodal user interfaces to improve hands-free interaction reliability in AR or VR devices.
Authors
Yuhong Zhang, Khanh An Nguyen, Xinyu Tian, KungYi Chang, Abraham Padua, Yuchen Xu, Ying Choon Wu, Gert Cauwenberghs, Tzyy-Ping Jung
Abstract
AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt their activities. GazeAgent identifies precise visual attention (PVA) events from multimodal streams of inertial measurement unit (IMU), gaze, and wearer-hand signals, validates their task relevance, and localizes functional targets as pseudo-calibration targets. In a 10-participant scripted study, the system detected all 820 predefined PVA opportunities and, under device disturbance, reduced median gaze error by 44.1% compared with explicit calibration. In a five-participant case study, autonomous calibration produced the lowest workload and interruption ratings, and all participants preferred it for routine use. By deriving task-critical targets from natural multimodal behavior, GazeAgent demonstrates the potential for continuous, adaptive, and low-burden gaze interaction on egocentric AI glasses.