Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
2026-07-09 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of recognizing how hands interact with objects in videos, noting that existing models often guess based on context rather than actual hand or object movements. They introduce a new training method that helps models learn from partial views of hands or objects and a special decoder that focuses on the locations and meanings of hands and objects. To test their approach, they created a new evaluation setup that isolates hand and object cues separately. Their model performs better than previous ones on this test and other related tasks, improving the understanding of hand-object interactions.
hand-object interactionvideo-language modelsmasked trainingembeddingdecoderaction recognitioninpaintingobject state recognitionrobot manipulationcue isolation
Authors
Masatoshi Tateno, Alexandros Stergiou, Risa Shinoda, Yoichi Sato, Dima Damen
Abstract
Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand-object masked training, which enables robust reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and object-centric embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific reasoning, we introduce Cue-Isolated HOI (CI-HOI), a new evaluation that assesses models' ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand- and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand- and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even robot manipulation action recognition, leading to more robust HOI understanding.