FeedbackTrack: Visual-Cortex-Inspired Cross-Frame Feedback for Transformer Tracking
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address how visual object tracking can be improved by adding a feedback loop inspired by the brain's visual cortex. Their method, FeedbackTrack, reuses intermediate information from previous frames to better guide the current frame's processing in Transformer-based trackers. This feedback happens in two ways—modulating queries and features—and requires very little extra memory or parameters. Tests on common tracking datasets show consistent improvements, mainly because the tracker benefits from using past information rather than just the current frame. They also found that feedback works differently depending on the layer depth in the Transformer architecture.
visual object trackingTransformertemporal integrationcross-frame feedbackquery modulationfeature modulationSPMTrackARTrackV2LaSOTGOT-10k
Authors
Yueyang Cang, Xiaoteng Zhang, Zhiyuan Ning, Yuchen He, Li Shi
Abstract
Visual object tracking requires effective temporal integration, yet most Transformer trackers still rely on predominantly feed-forward feature extraction. Existing temporal mechanisms typically update templates, prompts, queries, or prediction states, while intermediate representations are rarely reused to modulate corresponding processing stages. We propose \textbf{FeedbackTrack}, a visual-cortex-inspired framework that introduces sparse, group-level layer-aligned cross-frame feedback into pretrained Transformer trackers. Previous-frame intermediate states are detached, cached, and returned to corresponding Transformer groups in the current frame through two lightweight pathways: Query Feedback for token-level query modulation and Gate Feedback for context-dependent feature modulation. FeedbackTrack preserves the original tracking pipeline with only a fixed-size one-frame cache. Across SPMTrack and ARTrackV2, FeedbackTrack consistently improves five backbone configurations on LaSOT and GOT-10k, achieving 83.4 AO and 79.1 AUC with SPMTrack-G while adding less than 1\% parameters. Controlled comparisons show that cross-frame feedback outperforms same-frame modulation by 1.8--3.2 AO points, demonstrating that the gains mainly come from recurrent historical information. Further analysis reveals a non-uniform depth-dependent organization of learned feedback strengths, highlighting the effectiveness of recurrent feedback for Transformer tracking.