Summary
Tracking objects in videos using text help doesn’t always change how trackers work, especially when the visual quality gets worse. The authors created a test called DISTRACT that keeps the video and objects the same but changes the text instructions and image clarity to see how trackers respond. They found that some trackers use text instructions more deliberately, even when the images are blurry, while others hardly use the text at all. The study also showed when and where inside the tracker models the text has an effect, revealing early internal points that influence decisions without clear final benefits. This helps understand how text impacts object tracking when images are tough to see.
What this means in practice
- •For computer vision engineers: Identify specific tracker models and internal network stages where text input meaningfully changes tracking decisions, especially under degraded images.
- •For machine learning developers: Design or improve vision-language systems by understanding how and when textual cues affect model behavior and internal representations under challenging conditions.
Abstract
Vision-language trackers (VLTs) accept descriptions intended to assist tracking, but accepting text does not establish that its content changes tracking decisions, particularly as visual evidence deteriorates. We introduce DISTRACT, a matched-intervention benchmark that holds video, target, and distractor fixed while crossing four text priors with four image degradations along a controlled severity axis. We evaluate five VLTs on four datasets using behavioral contrasts, layerwise representation probes, and activation patching. Molmo2 and JointNLT exhibit directed text use. Some effects contract under degradation, whereas others show no significant endpoint change after multiplicity correction. UVLTrack, MMTrack, and DUTrack respond only weakly to changes in text semantics, with no directed behavioral effect surviving FDR correction in the tested settings. Model-internal analyses reveal that at decision-near states, correct-versus-conflicting text is decodable in Molmo2 and JointNLT but not in the other three systems. In the responsive models, decodability peaks at late states where replacement produces little behavioral change, whereas replacing selected earlier states redirects conflict-associated trajectories toward the target. Because matched-correct donors do not significantly outperform unrelated donors, these interventions identify earlier intervention-sensitive states without establishing target-specific semantic rescue. DISTRACT thus exposes when and where text affects tracking under degraded visual evidence.