Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study different ways to represent eye gaze data for use in vision models. They compare detailed raw gaze traces, simple event labels, and a new approach using angular movement during gaze events to see which is best for transferring knowledge across datasets. Their tests show that representing gaze as motion tokens linked to events can be more reliable depending on the dataset and direction of transfer. They also find that how gaze events are defined affects how well models trained on one dataset work on another. Overall, the authors suggest motion-based tokenization is a useful way to compactly describe gaze behavior for certain applications.
gaze representationevent-aligned tokensangular displacementvector quantizationtransfer learningegocentric gaze datasetsnext-token predictiontarget-domain regretmotion tokenizationevent construction
Authors
Virmarie Maquiling, Zhuojiang Cai, Enkelejda Kasneci
Abstract
Gaze is increasingly used as an input signal for vision and multimodal models, yet no consensus exists on how to represent it across datasets. Raw traces preserve detail but are noisy and device-dependent, while coarse event labels are easy to model but can discard local motion structure. We formulate event-aligned, fixed-horizon angular displacement as an interpretable, event-conditioned motion vocabulary and compare it with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations. To assess transfer alongside target predictability and token collapse, our evaluation combines next-token prediction with target-domain regret, low-order target references, paired bootstrap, order sensitivity, motif overlap, and frozen structural probes. In an event-aligned headset benchmark, angular-motion tokens have lower target-domain regret than frozen-codebook VQ tokens in one transfer direction, while the reverse direction is inconclusive. The probes reveal complementary representation properties, and event-only tokens show that low perplexity can retain little motion information. On a third egocentric dataset, a matched comparison of I-VT, native, and frame-span interfaces shows that event construction materially changes transfer: native events have the lowest regret into EGTEA, while frame-span events have zero motif overlap and fail severely as a source. Motion-based tokenization therefore provides a compact representation for event-aligned egocentric gaze streams, while the evaluation identifies how target predictability and event construction shape cross-dataset conclusions.