FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation
2026-07-20 • Robotics
Robotics
AI summaryⓘ
The authors developed a new robot method called FM-VLA that helps robots remember touch events to better handle tasks where what happened before matters a lot. Instead of relying mainly on images, their method uses force sensor data to create a simple memory of past contacts using a special model (VAE). This helps the robot understand tricky actions like pressing a button multiple times or finding hidden objects. Their approach works well on several tasks and is faster than older methods that only used vision. Overall, the authors show that adding force-based memory improves robot manipulation in complex situations.
Vision-language-action (VLA) modelsMemory-augmented modelsMarkovian assumptionForce sensorsVariational autoencoder (VAE)Contact-rich manipulationTemporal context reasoningRobotic manipulationInference overheadLatent representation
Authors
Ruicheng Li, Qixiu Li, Ruichun Ma, Yu Deng, Lin Luo, Zhiying Du, Jianfeng Xiang, Huizhi Liang, Ruicheng Wang, Jiaolong Yang, Baining Guo
Abstract
Vision-language-action (VLA) models have achieved impressive generalization in robotic manipulation, and recent memory-augmented VLAs have relaxed the Markovian assumption by conditioning on past images or language summaries. Vision-based memory approaches address this by conditioning on sampled past image frames, but they are computationally expensive and fundamentally limited when temporal events are visually ambiguous, e.g., pushing a button multiple times with small movements. We propose FM-VLA, a VLA model with force-based memory, enabling temporal context reasoning for non-Markovian, contact-rich manipulation. We encode force histories into compact force memory tokens with a variational autoencoder (VAE) pretrained with force time series reconstruction. By projecting force latent representations and short state history as additional conditioning tokens to the action expert module, we enable VLAs to leverage accumulated contact event history to guide manipulation. We evaluate FM-VLA on three memory-dependent tasks, including finding a hidden block, pressing a button, and wiping a dish for a specific number of times. Our lightweight force memory achieves over 80% success rate with minimal inference overhead, significantly outperforming baseline approaches. Project page: https://qft-333.github.io/FM-VLA-Page/