Zero-shot video tracking method enables automatic multi-object annotation
Towards Automatic Video Annotation with ASH: Zero-Shot Open-Vocabulary Multi-Object Tracking and Segmentation
Computer Vision and Pattern Recognition
Summary
Annotating objects in videos automatically is hard because current methods need lots of memory and can only process short clips or one category at a time. The authors created two new tools that let a video tracking system handle many object categories at once and work on long videos without retraining. One tool batches multiple text prompts efficiently during image processing, and the other extends tracking over longer sequences by matching object identities across smaller chunks of video. Their combined system works well on several benchmarks without training and uses less memory, making automatic video annotation more practical.
What this means in practice
- •For video annotation teams: Automatically label multiple objects in long videos without training on new data, reducing manual annotation effort.
- •For surveillance system developers: Implement memory-efficient multi-object tracking across lengthy video streams relying on open-vocabulary prompts without extensive retraining.
Authors
Arash Rocky, Q. M. Jonathan Wu
Abstract
Memory-attention-based Video Instance Segmentation (VIS) methods have demonstrated strong zero-shot tracking capability, yet their substantial memory requirements confine them to short video clips and their single-prompt inference design makes multi-category open-vocabulary tracking computationally prohibitive. This work introduces two contributions toward fully automated tracking annotation of arbitrary video. The Generalized Presence Token (GPT) reformulates SAM3's inference pipeline to process N text prompts simultaneously via virtual prompt batching, reducing image encoding cost from O(N) to O(1) with no modifications to any learned component. The Annotation and Segmentation Handler (ASH) extends any memory-attention VIS tracker to sequences of arbitrary length through overlapping temporal chunks with IoU-based inter-chunk identity matching, requiring no dataset-specific training. Instantiated on SAM3, the resulting pipeline -- SAM3-ASH -- achieves state-of-the-art HOTA on MOTS20 under fully zero-shot conditions and remains competitive with trained specialists across seven additional benchmarks, while peak GPU memory consumption stays below 25 GB, establishing a practical baseline for scalable, training-free automated video annotation.