Event driven refresh improves video object tracking accuracy and efficiency
Event-Driven Refresh and Recurrence Memory to Reduce Stale Grounding in Referring Video Object Segmentation
Computer Vision and Pattern Recognition
Summary
Tracking objects in video using natural language can get confused if objects disappear or change a lot over time. The authors improved an existing video object tracking system by adding a way to refresh its focus only when important changes happen in the video. This makes the system more accurate and reduces errors while using less computing power. Their tests show it works better than methods that refresh too often or too little, especially in longer or more complex videos.
What this means in practice
- •For video editing software developers: Integrate event-driven refresh to maintain segmentation accuracy in long videos with changing objects while reducing computational cost.
- •For security camera system engineers: Improve object tracking reliability in footage with occlusions or transient object appearances by dynamically refreshing object grounding based on detected changes.
Authors
Abu Hanif Muhammad Syarubany, Jaehyun Jang, Siwoo Lim, Seungyeon Ryu, Chang D. Yoo
Abstract
Referring Video Object Segmentation (RVOS) aims to produce a pixel-accurate mask sequence for an object specified by natural language. Sa2VA combines a multimodal large language model with SAM2 for grounded segmentation; however, its inference typically grounds the query from a small fixed set of initial keyframes and then relies on propagation. In long or dynamic videos, this can cause stale grounding and persistent false positives when the object composition changes (e.g., distractors enter or the target disappears/re-appears). We propose Event-Driven Refresh + Recurrence Memory (EDRRM), an enhancement that selectively re-invokes Sa2VA only at stable change points. EDRRM triggers refresh boundaries using an EMA-smoothed event score computed from tracking-derived cues (births/deaths and coarse composition/layout changes) with temporal constraints. A recurrence memory further retrieves anchor frames via CLIP similarity to re-condition the model on re-appearance events. Experiments on Ref-DAVIS17, MeViS, and ReVOS show that EDRRM achieves a competitive accuracy-efficiency trade-off relative to fixed-window and FrameDiff-SSIM baselines, maintaining comparable or superior J&F scores at substantially lower average refresh-call budgets and reducing false-positive failures. End-to-end runtime analysis further confirms that the overhead introduced by tracking, CLIP-based recurrence matching, and the identifiability gate remains modest relative to the dominant Sa2VA inference cost, thereby validating the efficiency of the proposed pipeline.