Reinforcement learning improves memory control for visual tracking models
SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking
Computer Vision and Pattern RecognitionMachine Learning
Summary
Tracking objects in videos relies on remembering past frames to keep following the right object over time, even when things get tricky like occlusions or distractions. Previous methods used fixed rules to decide which past frames to keep in memory, but these didn’t work as well on the latest tracking model called SAM 3. The authors teach a computer program using reinforcement learning to decide which frames to remember, improving tracking accuracy without slowing down the process. This approach works well on many video tracking tests and also boosts performance on the earlier SAM 2 model.
What this means in practice
- •For video analytics teams: Enhance object tracking reliability in video surveillance or monitoring systems by using learned memory control to handle occlusions and distractions better.
- •For augmented reality developers: Improve real-time tracking of objects in AR applications through smarter memory management for more stable and accurate overlays.
Authors
Tomáš Čížek, Matej Straka, Illia Volkov, Matěj Kripner, Ervin Macić, Jiri Matas, Klara Janouskova, Martin Schmid
Abstract
Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.