Smart glasses improve video answers by focusing on gaze and context
GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction
Computer Vision and Pattern Recognition
Summary
When smart glasses watch what you do, they need to answer questions about the video without using too much computing power. The authors created GazeLens, a system that picks important video frames and looks at where you are actually looking (your gaze) to keep details that matter. Instead of keeping every tiny part of the video, GazeLens combines focused spots with the wider scene to keep just the right information. This helps it answer questions better and faster than other methods.
What this means in practice
- •For augmented reality developers: Create smarter AR glasses that quickly answer user questions by focusing on where they look and surrounding context to save processing power.
- •For assistive technology designers: Build wearable devices that help people understand scenes by efficiently summarizing relevant video based on gaze and surrounding details.
Authors
Trong-Thang Pham, Anh Nguyen, Hien Van Nguyen, Ngan Le
Abstract
Smart glasses can help users understand their activities and surroundings, but small vision-language models must preserve useful evidence within limited computation. This is difficult because redundant frames consume the visual budget, while aggressive spatial compression can discard relevant details. Although gaze identifies the wearer's focus, the question may require evidence from the surrounding scene. To address these constraints, we propose GazeLens, which combines frame selection with gaze- and question-guided token reduction. First, HOG transitions and image sharpness select frames before visual encoding. For each selected frame, Gaussian pooling then forms gaze-focused and complementary summaries. Because their relevance depends on the question, a lightweight gate blends them into a primary token. Together with a global scene token, this retains both focused and contextual summaries within two tokens per frame. On StreamGaze, GazeLens achieves higher macro and micro accuracy than the evaluated full-grid and token-reduction baselines, with lower input-forward FLOPs than the shared-backbone token-reduction baselines.