Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking
Computer Vision and Pattern Recognition
Summary
The authors developed LVTrack, a new method for tracking a single object in a video using language descriptions. They point out that language helps start the tracking but can cause errors if used too much later on. Their approach uses a special transformer that controls how language is applied during different parts of tracking, reducing mistakes and keeping the system efficient by relying on a pretrained vision-language model. They also improved tracking accuracy with smarter position encoding and memory features. Tests show their method works well on common tracking benchmarks.
Referring single-object trackingVision-language pretrainingTransformerSemantic driftGated Feature InjectorPositional encodingAutoregressive predictionGaussian-smoothed KL lossMemory mechanism
Authors
Han Wang, Yuxuan Liu, Yuhan Sun, Jian Yang, Xiaotong Xu, Yixuan Lv, Zhuang Zhou, Shengyang Li
Abstract
Referring single-object tracking enables language-grounded target initialization and subsequent tracking by jointly leveraging semantic cues and visual templates. The core difficulty is to use language differently across stages: it is indispensable for grounding but can induce semantic drift during tracking when overemphasized. Meanwhile, current methods often require costly vision-language alignment training. We present LVTrack, a pure transformer framework that introduces a mode-conditioned Gated Feature Injector to adaptively regulate textual guidance and alleviate semantic drift. Together with targeted adaptations, it directly harnesses a frozen vision-language pretrained model, greatly reducing training cost and preserving strong language understanding. To further improve temporal localization, LVTrack integrates hybrid relative-absolute positional encodings with a lightweight memory mechanism and optimizes autoregressive box prediction using a Gaussian-smoothed KL loss. Extensive experiments on standard benchmarks demonstrate that LVTrack achieves strong performance.