4D scene graphs maintain object identity across occlusions using memory
Cross-Visibility-Gap Identity Persistence for 4D Scene Graphs
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
This paper addresses how to keep track of objects in a 3D scene over time, even when they temporarily disappear inside other objects and then reappear. The authors develop a method that remembers the relationship between a hidden object and the object carrying it, without relying on appearance or motion clues. Their system uses a special memory structure to store and retrieve this relational information, successfully reconnecting objects after they reappear. Tests show their method works much better than simpler approaches and generalizes to longer unseen gaps.
What this means in practice
- •For robotic system builders: Improve object tracking in robots when objects are temporarily hidden inside others by storing relational memory of object-to-carrier bindings.
- •For augmented reality developers: Enhance AR scene understanding by persistently tracking objects through occlusions using relational information rather than just visual appearance.
Authors
Jiahui Hu, Li Gao, Yu Liu, Pengzhan Zhou
Abstract
4D scene graphs require persistent object identity when an object disappears inside an opaque carrier and later reappears as a fresh detection. When appearance and target-centric motion are unavailable, identity recovery must rely on relational history. We formulate cross-visibility-gap identity persistence as historical-node recovery through a remembered target--carrier binding. We introduce CVG-SGM, a perception-frontend-compatible transfer--maintain--read memory. Geometry-weighted cross-node interaction transfers target-specific evidence into carrier states, a shared per-node GRU maintains this information during transport, and a two-time readout combines stored carrier--identity compatibility with current detection--carrier geometry to recover the historical identity. In the controlled one-identity-per-carrier setting, CVG-SGM achieves 1.000 re-link accuracy, compared with 0.570 for both no-memory and parameter-count-matched no-write controls. It also generalizes to scene-disjoint, unseen longer visibility gaps. On real RGB-D/SAM2 feature-level episodes, it correctly re-links 36/36 held-out detections, compared with 28/36 for the strongest fixed carrier--identity control. Counterfactual carrier-memory swaps redirect predictions according to the swapped bindings, supporting the role of stored relational information in identity recovery. The current approach remains limited by shared-carrier identity ambiguity and does not provide end-to-end raw-video re-identification.