AI summaryⓘ
The authors created Event ActivityNet, a large dataset for studying long-term event-based action recognition, since most existing datasets only have short clips. Their dataset uses simulated event data from untrimmed ActivityNet videos and includes many action classes with detailed timing and captions. They test different ways to analyze this data and show that training with progressively larger video scales improves accuracy in recognizing and localizing actions over time. They also find that pretraining on Event ActivityNet before fine-tuning on native event data leads to better results. This benchmark aims to help future research on understanding long-term actions from event data, although real camera data is still needed for final real-world use.
event-based action recognitionActivityNetevent voxelstemporal action localizationpretrainingevent-language alignmentevent framingLPIPSRGB-event data
Authors
Cheng-Yao Hong, Ting-Wei Lin, Yun-Chung Lai, Hua-Wei Lee, Hwann-Tzong Chen, Tyng-Luh Liu
Abstract
Long-horizon event-based action understanding remains underexplored because existing datasets largely comprise short, trimmed clips, while collecting native event streams with dense temporal annotations is costly. We introduce Event ActivityNet, a large-scale simulated-event benchmark derived from human-annotated, untrimmed ActivityNet videos. It comprises 3,263 videos, 200 action classes, and 106.94 hours, with matched 5-bin and 9-bin event-voxel representations, temporal action annotations, and timestamped captions. The benchmark supports annotated-segment action recognition, auxiliary event-language alignment, and causal online temporal action localization. We generate event voxels directly from non-interpolated source videos in decoded frame order, retain per-video rational nominal or average frame-rate metadata for approximate time mapping, and use action-center reconstruction LPIPS as a soft diagnostic of retained reconstructable content. We establish baselines for adaptive event framing, prompt-caption alignment, and event-only, RGB-only, and RGB-event localization. Under a progressive nested-scale training protocol, recognition Top-1 accuracy increases from 52.25 to 66.42, while online temporal localization average mAP improves from 21.7 to 29.0. Moreover, staged Event ActivityNet pretraining followed by native-event fine-tuning consistently outperforms target-only and joint-from-scratch training across multiple supervision budgets. Event ActivityNet provides a scalable benchmark for long-horizon event modeling, although native-camera evaluation remains essential for deployment-oriented conclusions.