Cross view system tracks and captions people in videos consistently

CRTrack-Cap: A New Benchmark for Cross-View Dense Video Object Captioning

Computer Vision and Pattern Recognition

Summary

Tracking people from videos taken by multiple cameras is hard because each camera shows its own view. This paper addresses this by linking the tracks of the same person across cameras and writing one combined description for that person using all views. The authors created a dataset named CRTrack-Cap with videos from multiple cameras, corrected text links, and added clear captions for people. They also made a method that groups tracks and combines caption information to make one reliable description per person. Their system works better than previous single-camera methods when tested on different scenes.

What this means in practice

  • For video surveillance teams: Enable consistent tracking and describing of individuals across multiple camera views for better scene understanding and reporting.
  • For smart city operators: Combine video data from various cameras to generate unified person descriptions, improving monitoring and analytics across public spaces.

Authors

Xiangqun Zhang, Likai Wang, Zekun Qian, Rui Zhou, Ruize Han, Wei Feng

Abstract

Dense video object captioning (DVOC) localizes, tracks, and describes object trajectories within a video. Existing methods process each camera view independently, producing separate camera-local tracklets and captions for the same person across cameras. These outputs lack a unified description and leave complementary appearance evidence across views unused. We therefore formulate cross-view DVOC (CVDVOC), which associates these tracklets into scene-level global trajectories and generates one caption for each trajectory from synchronized views. To support this task, we construct CRTrack-Cap from CRTrack by auditing text-identity links and converting usable referring expressions into canonical captions. It retains 425 global trajectories and 838 local tracklets across 13 scenes, with canonical captions for 335 identities. We adapt CHOTA to the cross-view setting as CV-CHOTA to jointly evaluate tracking and caption quality. Our reference framework combines View-Exclusive Cross-Camera Association (VECA) for grouping tracklets with Conflict-Evidence Gating (CEG) for integrating caption evidence using its reliability and disagreement across views. Against three single-view methods adapted with MvMHAT for cross-view association, our framework achieves CV-CHOTA scores of 83.43 and 33.48 on the scene-disjoint in-domain and cross-domain tests, respectively. Together, CRTrack-Cap and our framework provide a benchmark and a strong baseline for describing the same person consistently across camera views.