EgoActionBench advances precise understanding of hand actions in egocentric videos
EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Human Videos
Computer Vision and Pattern Recognition
Summary
Watching videos from a person's point of view to understand what their hands are doing is tricky because actions are fast, detailed, and often overlap. The authors created EgoActionBench, a large collection of videos with detailed labels about what each hand is doing and when. They also made a new way to judge how well computer models can understand these hand actions precisely by timing and content. Testing many models revealed current systems struggle with fine details about hand use and timing. Their work shows an important step toward machines that understand human actions better for use in everyday tasks.
What this means in practice
- •For robotics developers: Develop robots that recognize and imitate precise hand actions from wearable cameras for improved interaction and task execution.
- •For augmented reality developers: Create AR systems that accurately interpret hand actions in real time from a user's viewpoint to enable intuitive controls.
Authors
Yujiao Shen, Jiahe Song, Haitian Li, Jingkang Yang, Ziwei Liu
Abstract
Learning from human data offers a scalable path toward embodied intelligence, yet it requires models to accurately understand fine-grained human actions from egocentric observations. However, existing benchmarks remain limited in action atomicity, temporal precision, and hand-action association. We introduce EgoActionBench, a benchmark that represents continuous human manipulation as temporally localized atomic actions associated with the left hand, right hand, or both hands. EgoActionBench contains 1,407 episodes with 41,132 human-refined action annotations. To evaluate such dense predictions, we propose an evaluation protocol that retrieves temporally eligible candidates, selects a prediction by content agreement, and scores semantic fidelity and temporal overlap alongside action precision and recall, jointly capturing what happens, when it happens, and which hand performs it. We benchmark 20 baseline models and systems on EgoActionBench and further adapt two backbones, Qwen3-VL-4B and LLaVA-OneVision-2-8B, through supervised fine-tuning followed by reinforcement learning on automatically annotated training episodes. EgoAction improves EgoActionScore by 15.9 and 23.2 points on the two backbones, and both adapted models outperform all open-source baselines. Our experiments reveal substantial limitations of current models in action atomicity, temporal precision, and hand-action association, highlighting the gap between general video understanding and the action understanding required for embodied intelligence.