Summary
Understanding detailed traffic behaviors requires recognizing what several agents are doing simultaneously and how their actions relate to the road layout. The authors present Action-Slot, a new method that breaks down complex scenes into predefined activity categories and processes them holistically over time. This approach helps detect overlapping activities and distinguish active from inactive agents using video alone. Their model also learns useful spatial and temporal features that enable locating actions without detailed labels, supported by a new synthetic dataset named TACO. Tests show this method accurately recognizes and localizes traffic activities, even transferring well from simulated to real-world data.
What this means in practice
- •For autonomous vehicle teams: Detect and identify complex multi-agent traffic behaviors from video to improve situational awareness and planning.
- •For traffic monitoring operators: Localize and recognize concurrent traffic activities with minimal annotation effort using weakly supervised models.
Authors
Yu-Ho Chang, Chi-Hsi Kung, Yi-Hsuan Tsai, Yi-Ting Chen
Abstract
Atomic activity understanding aims to recognize and localize structured traffic behaviors that jointly encode motion patterns and their grounding in road topology. Unlike conventional action recognition, atomic activities are multi-agent, multi-label, and topology-aware: multiple activities co-occur while many agents remain inactive. We introduce Action-Slot, a structured action-centric representation learning framework. Slot attention is widely used for object-centric decomposition, but its permutation-invariant design and object-level inductive bias are misaligned with atomic activity semantics. We reformulate slot learning as structured activity decomposition through three designs: (1) category-aligned action slots that anchor slots to predefined activity categories, (2) parallel spatio-temporal slot updating for holistic video-level reasoning, and (3) background and negative-slot regularization that enforces competition between foreground activities and irrelevant regions. Together these establish an activity-centric inductive bias that disentangles concurrent and asynchronous activities directly from raw video. Beyond recognition, the learned representations encode transferable spatio-temporal grounding signals. We further propose an attention-difference-based pseudo mask selection framework that suppresses false positives by measuring attention changes before and after candidate region removal, enabling weakly supervised localization without dense annotations. To support systematic evaluation, we introduce TACO, a balanced synthetic dataset with full atomic activity coverage and pixel-level annotations. Experiments on OATS, TACO, and annotated nuScenes show superior recognition, strong sim-to-real transfer, and state-of-the-art weakly supervised localization.