CoDecide speeds up video analytics by decoding only new content

CoDecide: Content-Adaptive Video Decoding for High-Throughput Video Analytics

Computer Vision and Pattern Recognition

Summary

Video analysis can be slow because every video frame needs to be decoded before processing. The authors developed CoDecide, a system that detects only the frames containing new or changed content by analyzing the compressed video data. This method avoids decoding unchanged parts, saving a lot of time while still finding most moving objects. It also works on new camera setups without extra training. CoDecide can run 15 to 24 times faster than traditional methods that decode every frame.

What this means in practice

  • For video surveillance teams: Process archived surveillance videos faster by decoding only frames with new activity, reducing compute costs while retaining important object detections.
  • For smart city infrastructure engineers: Enable real-time traffic and public safety monitoring by selectively decoding video frames with novel events, improving efficiency in large-scale camera networks.

Authors

Shristy Shah, Thomas B. Moeslund, Kamal Nasrollahi

Abstract

Analytics on video archives are bottlenecked by decoding. In a dense pipeline that runs a detector on every frame, decoding is a small fraction of the runtime, but once the detector is restricted to selected frames, decoding takes over as the main cost. We introduce CoDecide, a retrospective analytics system that only decodes the frames where new content appears, using signals from the compressed bitstream to find them. Motion tells us something is moving, but not whether it is new to the scene or has been there the whole time. To make that distinction, we supervise a lightweight compressed-domain student on a label-free novelty signal from a frozen DINOv2 teacher, which fires on semantically new regions rather than on any motion. Consequently, the student model generalizes to new camera deployments without requiring per-camera retraining. At inference, it decides which frames to decode and which regions to reconstruct. Detections from the decoded frames are propagated to the skipped frames using motion vectors. We evaluate CoDecide with a leave-one-dataset-out protocol on VIRAT, MEVA, and ShanghaiTech. Against a dense per-frame detector, CoDecide recovers more moving-object detections than prior compressed-domain selectors at an equivalent decoding budget, outperforming even a competitor trained directly on the target camera. CoDecide runs 15 to 24x faster than a pipeline that decodes and detects every frame. Code is available at https://github.com/shristy51/CoDecide.