Hierarchical memory and tools improve reasoning in self driving cars
Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving
Computer Vision and Pattern RecognitionRobotics
Summary
Self-driving cars use large vision-language models to understand and navigate roads, but these models often make mistakes and forget what they just saw. The authors designed a system that helps the car remember short-term details about the current scene and also learn from long-term past driving experience. Their system uses tools during decision-making to check and improve its answers, and it remembers good strategies for future trips. This approach improved the car’s ability to reason and answer questions about driving scenarios better than previous methods.
Vision-Language ModelsHierarchical MemoryAutonomous DrivingTool InvocationReasoningShort-term MemoryLong-term MemoryCoT ExplanationsSpatio-temporal PerceptionSelf-evolution
Authors
Baojie Chen, Zijun Jia, Jing Zhong
Abstract
VLMs have shown promise for autonomous driving, yet still suffer from hallucination, weak spatio-temporal perception, and limited generalization. Recent methods improve reasoning and decision-making through CoT explanations, retrieval-augmented generation or the static injection of tool outputs. Although these mechanisms enrich the context, the model neither proactively perceives scene information nor accumulates experience after answering. To overcome these limitations, we present, to our knowledge, the first synergistic framework that tightly couples hierarchical memory with proactive tool invocation in a closed reasoning loop. Our contributions are threefold. (i) Hierarchical Driving Memory: a scene-level short-term memory maintains the dynamic scene state, and an evolving long-term memory retrieves reusable experience and tool strategies. (ii) Memory-Tool Synergistic Reasoning Framework: guided by the scene state and retrieved experience, the model adaptively invokes tools to refine its reasoning at inference time and consolidates reusable experience into a long-term memory pool offline. (iii) Data Generation and Two-stage Training Pipeline: verified memory-tool trajectories built by multi-step teacher rollout are used to train with SFT and GRPO. Our 7B model reaches an overall reasoning score of 80.03 and MCQ accuracy of 79.09% on DriveLMM-o1, surpassing the strongest baseline by 7.74 MCQ points and generalizes strongly across benchmarks. Notably, ablation and analysis studies validate the effectiveness of each component and further reveal the complementary roles of hierarchical memory. Short-term memory strengthens spatio-temporal understanding, improving STSBench accuracy by 24.2 points, while offline long-term memory consolidation yields an additional 3.57-point MCQ gain with all parameters frozen, demonstrating continual self-evolution through accumulated driving experience.