ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on a task where a video object must be segmented based on a natural language description. They improve existing methods that only make one initial guess about the object's location and then try to track it through the video without checking if the guess was good. Their new system, ReflexTrack, uses a feedback loop to refine these guesses both for individual frames and across the whole video, fixing mistakes without needing new training. This approach leads to more reliable segmentations on test datasets, showing better performance than previous training-free methods.
Referring Video Object Segmentation (RVOS)multimodal large language modelspromptable segmentation modelsspatial groundingmask propagationbounding box refinementfeedback loopmask reflectiontraining-free inferencevideo object tracking
Authors
Yuanjia Li, Tianyang Xu, Tao Zhou, Zhangyong Tang, Xiao-Jun Wu, Josef Kittler
Abstract
Referring video object segmentation (RVOS) requires segmenting a target specified by natural language throughout a video. Recent agentic approaches combine multimodal large language models with promptable segmentation models to perform RVOS without task-specific training. However, most pipelines rely on one-shot spatial grounding followed by mask propagation, leaving both the initial prompts and temporal predictions largely unverified. We introduce ReflexTrack, a training-free, feedback-driven agent that closes this loop at both spatial and temporal levels. Mask-guided Spatial Refinement evaluates the mask induced by the current keyframe prompt and iteratively updates the bounding box together with positive and negative points, yielding a more reliable initialization. Video-level Mask Reflection assesses the complete mask sequence, localizes unreliable intervals, selects complementary repair keyframes, and generates candidate predictions through mask-guided re-propagation. Only candidates that provide a verified improvement are used to update the affected intervals, preserving reliable predictions elsewhere. All components remain frozen during inference. ReflexTrack achieves an overall $\mathcal{Q}$ score of $69.7$ on Ref-VPS and a $\mathcal{J}\&\mathcal{F}$ score of $67.2$ on ReasonVOS. These results demonstrate that prediction-level feedback substantially improves the reliability of training-free RVOS.