Language queries improve multi-camera tracking of multiple targets
Refer-CityFlow and M3Track: A Benchmark Dataset and Reference Framework for Language-Driven Multi-Target Multi-Camera Tracking
Computer Vision and Pattern RecognitionMultimedia
Summary
Tracking specific people or objects in video can be done by describing them in words. This work focuses on tracking multiple targets across several cameras that have little overlap in their views. The authors created a new dataset called Refer-CityFlow with many video clips, cameras, and user-written language queries to test this challenge. They also built M3Track, a system that uses a large language model to combine detection, tracking, and matching across cameras, performing much better than existing methods.
What this means in practice
- •For video surveillance teams: Track multiple people or vehicles across different cameras using natural language descriptions to improve monitoring in complex scenes.
- •For smart city developers: Develop city-scale tracking systems that link objects across many roadside cameras based on textual queries for better traffic management.
Authors
Phuong T. Nguyen, Hao Vo, Tam Bang, Hoang K. Tran, Hoang-Minh Dau, Mahshid Malazizi, Tuan-Anh Hoang, Mina Sartipi, Thanh-Nam Doan, Ngan Le, Hoang H. Nguyen
Abstract
Referring multi-object tracking enables users to specify targets through natural language, but existing Refer-MOT benchmarks primarily focus on single-camera settings. We formalize Referring Multi-Target Multi-Camera Tracking (Refer-MTMCT), where free-form queries specify identity-consistent trajectories to be recovered across a network of sequential, sparse-overlap cameras. We introduce Refer-CityFlow, a benchmark built on CityFlow's global cross-camera identity annotations and augmented with language queries requiring joint language grounding, cross-camera association, and spatio-temporal reasoning. Refer-CityFlow covers 96 videos from 40 roadside cameras with 660+ identity-consistent cross-camera trajectories, 4,362 trajectory-level descriptions, and 7,699 queries, including 1,699 real-user queries collected from 50 participants. Reference experiments on the four-camera Scenario 02 show that single-camera referring trackers reach 14.95-18.28 HOTA on real-user queries, while the evaluated cross-view method achieves about 0.05 HOTA in direct transfer. To establish reference performance, we provide M3Track, a modular multimodal reference framework in which a large language model (LLM)-based agent orchestrates interchangeable detection, tracking, and cross-camera association modules. M3Track reaches 44.10 HOTA and 65.40 IDF1 on the same real-user query set. The project repository is available at https://github.com/phuongnguyen194/Refer-CityFlow-and--M3Track.