Lifecycle-aware memory improves multi-object tracking with SAM2 models
LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT
Computer Vision and Pattern Recognition
Summary
Tracking multiple objects in videos can get confused when objects are close together, disappear, or reappear after hiding. The authors rethink this problem as keeping good memory of each object’s history. They build a system called LiAM-SAM that carefully manages object information through their whole lifecycle, fixing mistakes like duplicate tracking or mixing up identities. Their approach helps machines track objects more accurately and reliably, especially in crowded or challenging scenes.
What this means in practice
- •For video surveillance teams: Track multiple people or vehicles more reliably in crowded or complex scenes using lifecycle-aware memory management in SAM2-based tracking systems.
- •For autonomous vehicle developers: Improve long-term tracking of nearby cars and pedestrians to prevent identity confusion during occlusions and close interactions.
Authors
Grégoire Francisco, Alessandro D'Amico, Samuele Costantini, Gianpiero Francesca, Lorenzo Garattoni
Abstract
Segmentation-based multi-object tracking (MOT) with foundation video models such as SAM2 offers strong localization quality, yet remains fragile in crowded, real-world scenes. In detector-prompted SAM2 pipelines, failures typically arise at three stages of the object lifecycle: (i) erroneous or duplicate track initiation, (ii) memory drift during close interactions, and (iii) unreliable re-identification after long occlusions or re-entry. These errors corrupt object memory and accumulate over time, making long-horizon tracking unstable. In this paper, we reframe MOT as a lifecycle memory integrity problem. We present LiAM-SAM, a Lifecycle-Aware Memory (LiAM) framework with targeted mechanisms for each of the three failure modes. At track birth, to prevent faulty or duplicate initiations, we apply contrastive track initiation, which conditions each prompt on existing nearby tracked instances. To preserve memory integrity during strong interactions, we introduce motion- and geometry-grounded memory correction that resolves interaction confusions and suppresses drift. For reliable re-identification after disappearance, we maintain an adaptive context memory that promotes diverse and trustworthy references as long-term identity anchors. Finally, similarity aware spatial pruning optionally selects the memory tokens to retain at cross-attention time, improving efficiency with minimal accuracy loss. LiAM-SAM represents a modular, detector-agnostic, SAM2-based MOT system that achieves state-of-the-art HOTA and IDF1 on the evaluated benchmarks. In association-challenging environments, our ablations show that LiAM improves a detector+SAM2 baseline by +10.5 HOTA, +17.4 AssA, and reduces identity switches by 96%.