Visual language helps fix tracking errors in video object segmentation
When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation
Computer Vision and Pattern Recognition
Summary
Tracking objects in videos can go wrong when things block the view or objects look similar, causing the system to lose track of the right target. The authors present Dual-Agent, a method that can recognize when tracking goes wrong and then fix it using a language-assisted verification process. One agent checks if the tracking is still accurate by looking at the scene and language cues, and if a problem is found, a second agent helps recover the correct object before continuing. This approach improves tracking accuracy on several benchmarks and works well even with different tracking systems without retraining.
What this means in practice
- •For video analytics developers: Improve robustness of video object tracking systems by adding explicit failure detection and automatic recovery to handle occlusions and distractions.
- •For autonomous vehicle software teams: Enhance reliability of object tracking in driving scenarios by integrating verification agents that detect tracking errors and correct identity switches.
Authors
Valay Bundele, Susmit Agrawal, Mehran Hosseinzadeh, The Nam Nguyen, Hendrik P. A. Lensch
Abstract
Recent trackers built on foundation models such as the Segment Anything Model (SAM) achieve strong performance by propagating object masks through memory. However, this strength is also a key source of failure: under occlusion, reappearance, or distractors, an incorrect prediction can enter the tracking loop and be propagated over time, resulting in persistent identity drift. Existing approaches seek to prevent such failures through improved memory and temporal modeling, but once a failure occurs, the tracker has no explicit mechanism to recognize or correct it. We address this gap with Dual-Agent, a failure-aware tracking framework that complements propagation with explicit verification and recovery. We address this gap with Dual-Agent, a failure-aware tracking framework that complements propagation with explicit verification and recovery. A vision-language verification agent first reasons over spatio-temporal context to determine whether the tracker is still following the target. When a failure is detected, a correction agent uses temporal context to recover the target before tracking continues. The correction agent is trained on real SAM3 failures, augmented with targeted identity-switch and distractor perturbations that expose it to challenging ambiguities during recovery. On LVOS-v2 and SecVOS, Dual-Agent improves SAM3 by 1.2 and 2.7 $J\&F$, respectively. Importantly, the proposed framework transfers without retraining to SAM2.1, SeC, and Cutie, as well as to long-term video object tracking benchmarks including LaSOT-ext, TNL2K, and VOT-LT2022. Our results show that explicitly reasoning about when tracking fails and how to recover provides a complementary path to robust tracking beyond improving propagation alone. Project page: valaybundele.github.io/Dual-Agent.