Video transformers trimmed to speed up collision prediction with little accuracy loss
Investigating Spatiotemporal Redundancy in Video Transformer for Collision Anticipation
Computer Vision and Pattern Recognition
Summary
Collision anticipation systems use video transformers to watch workers and equipment and predict accidents, but these systems can be slow and costly to run. The authors studied where in the video transformer the computer is doing extra work that doesn’t improve predictions much. They found that many video frames over time are very similar and can be combined without losing much accuracy. Also, parts of the model’s neurons can be removed carefully to keep performance strong. These tricks together make the system almost twice as fast without hurting its ability to predict collisions.
What this means in practice
- •For mobile robot developers: Reduce computing time and power for collision anticipation models on mobile robots by merging temporal video tokens and pruning neural units without losing accuracy.
- •For construction safety teams: Deploy faster, energy-efficient video-based collision prediction systems in construction sites by applying temporal compression and neuron pruning to monitor worker-equipment proximity.
Tested on one dataset.
Authors
Xiaoshan Zhou
Abstract
In worker-equipment proximity monitoring, video transformers are widely used for collision anticipation and have demonstrated strong performance. However, their accuracy comes with substantial computational demands, creating a tension with the need for low-latency inference on mobile robots and the pursuit of lower-carbon computation in construction. To address this, this study investigates where computation within an established video transformer is redundant and whether that redundancy can be removed without materially degrading predictive performance. Using VideoMAEv2-Base on the Nexar Collision Prediction dataset, we first examine how collision-relevant information evolves across network depth and then investigate two complementary forms of redundancy: structured capacity redundancy in multilayer perceptrons (MLPs) and spatiotemporal redundancy in the token stream. Linear probes show that interpretable motion cues, including flow magnitude, looming, and approach versus retreat, are most accessible at intermediate layers, whereas collision-label discrimination strengthens toward the final layer. Token redundancy is axis-specific: adjacent temporal-token similarity reaches 0.970 in later layers, while spatial similarity falls to 0.297, indicating substantially greater redundancy across time than across space. Exploiting this asymmetry, temporal token merging reduces backbone computation from 356.99 to 178.50 GFLOPs and latency from 12.69 to 7.21 ms per clip, a 1.76x speedup, while mean average precision changes only from 0.7478 to 0.7443. Importance-guided retention of 50% of MLP units preserves an AUC of 0.753, compared with 0.529 under matched random retention, and reveals that pruning alters score calibration before discriminative ranking collapses. These findings establish a new pathway for pursuing faster algorithms through targeted temporal token compression and neuron pruning.