Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking
Computer Vision and Pattern Recognition
Summary
The authors developed LVTrack, a new method for tracking a single object in a video using language descriptions. They point out that language helps start the tracking but can cause errors if used too much later on. Their approach uses a special transformer that controls how language is applied during different parts of tracking, reducing mistakes and keeping the system efficient by relying on a pretrained vision-language model. They also improved tracking accuracy with smarter position encoding and memory features. Tests show their method works well on common tracking benchmarks.
Authors
Han Wang, Yuxuan Liu, Yuhan Sun, Jian Yang, Xiaotong Xu, Yixuan Lv, Zhuang Zhou, Shengyang Li
Abstract
Referring single-object tracking enables language-grounded target initialization and subsequent tracking by jointly leveraging semantic cues and visual templates. The core difficulty is to use language differently across stages: it is indispensable for grounding but can induce semantic drift during tracking when overemphasized. Meanwhile, current methods often require costly vision-language alignment training. We present LVTrack, a pure transformer framework that introduces a mode-conditioned Gated Feature Injector to adaptively regulate textual guidance and alleviate semantic drift. Together with targeted adaptations, it directly harnesses a frozen vision-language pretrained model, greatly reducing training cost and preserving strong language understanding. To further improve temporal localization, LVTrack integrates hybrid relative-absolute positional encodings with a lightweight memory mechanism and optimizes autoregressive box prediction using a Gaussian-smoothed KL loss. Extensive experiments on standard benchmarks demonstrate that LVTrack achieves strong performance.