Episodic memory improves navigation with vision and language in 3d environments
MemVLN: Streaming Episodic Memory for Vision-Language Navigation
Computer Vision and Pattern Recognition
Summary
Following instructions to move around in 3D spaces is hard because the agent needs to remember where it has been. The authors introduce MEMVLN, which helps agents keep a small but useful memory of what they saw and did while moving. This memory helps agents do better on tests where they follow language directions to navigate. It also works better than previous methods when tested in a real room. The memory learns to summarize routes so the agent can recall and follow them more accurately.
What this means in practice
- •For robotics engineers: Build robots that follow spoken or written directions more reliably by using memory to recall prior observations and movements.
- •For augmented reality developers: Create AR guides that better track user locations over time to direct navigation in complex indoor spaces.
Authors
Yan Wang, Jiangyong Huang, Baoxiong Jia, Siyuan Huang
Abstract
Vision-language navigation (VLN) requires an embodied agent to follow language instructions through 3D environments. Maintaining persistent episodic memory remains a challenge for VLM-based streaming agents in long-horizon navigation. We present MEMVLN, a streaming episodic memory framework that maintains a bounded bank of compressed perceptual features annotated with timesteps and action-derived relative poses, and produces compact memory tokens via cross-attention. We pair the proposed memory with a frozen pretrained geometry encoder. To learn the memory module, we design a masking-based trajectory summarization task with multi-granularity sub-trajectory decomposition, encouraging route reconstruction exclusively from episodic memory. Under navigation training restricted to the official R2R/RxR trajectories, MEMVLN achieves state-of-the-art performance on most reported metrics on standard VLN-CE benchmarks. Long-horizon analyses further characterize when episodic memory is useful. For physical evaluation, we collect 30 instruction-conditioned episodes in one physical scene, use 24 as demonstrations for post-training, and hold out six for testing. On the held-out episodes, MEMVLN improves success over the VLM baseline from 17% to 40%, and post-training further raises it to 47%.