Object centric 4D memory improves long horizon robot manipulation success
OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation
Robotics
Summary
Some robots need to remember objects that are out of sight or have been moved to complete long tasks. The authors created OCC4M, a system that keeps track of objects in space and time, including their movement and what they contain. This helps robots decide what to do next without remembering every single frame of video. Their tests show OCC4M works much better than previous methods, especially when the point of view changes or after many steps.
What this means in practice
- •For robotics engineers: Improve robots' ability to manipulate objects over many steps by using a memory that tracks object states explicitly over time and space.
- •For warehouse automation teams: Enhance robotic systems to remember and reason about moved or hidden items during multi-step picking and packing tasks.
Authors
Jack B. Jedlicki, Tanguy Dieudonné, Heng Yang
Abstract
Long-horizon manipulation often requires reasoning about state absent from the current view, such as a vanished object's location, temporal identity, or the contents of a shuffled container. We present OCC4M ("Occam"), an object-centric 4D memory that maintains persistent tracks in a shared world frame and explicitly represents temporal, motion, and containment relations. A vision-language model (VLM) queries this structured memory to select actionable targets for history-free low-level execution. Across seven simulation conditions and 350 episodes, OCC4M achieves 96.6% memory success and 88.9% end-to-end success, versus 54.6% and 57.7% for FrameSamp, a raw-history VLM baseline using Gemini 3.7 Flash with the complete observation history and the same executor. In a controlled viewpoint-transfer test, OCC4M maintains 100% memory and 98% end-to-end success after a viewpoint change, while full-history FrameSamp falls to near-zero success. On 20 fixed-camera Franka episodes, OCC4M reaches 85% joint memory accuracy, versus at most 30% for FrameSamp across context sizes from $K=16$ to the complete history, and completes 45% of full two-stage tasks. These results support explicit object-centric memory for persistent spatiotemporal reasoning in long-horizon manipulation. Qualitative videos are available at https://occ4m-sup.github.io/occ4m-supplementary/.