Autonomous driving system improves decision making with memory and tools
Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving
Computer Vision and Pattern RecognitionRobotics
Summary
Driving AI systems often struggle to understand scenes fully and remember past experiences, leading to mistakes. The authors designed a new method that helps the AI remember recent and long-term driving information while actively using helpful tools during its reasoning process. This approach allows the system to learn from experience and improve over time, leading to better scene understanding and safer driving decisions. Their tests show notable improvements in driving reasoning and question answering compared to previous methods.
What this means in practice
- •For autonomous vehicle engineers: Improve self-driving cars’ situational awareness and decision-making by integrating active memory and tool use for better real-time responses.
- •For robotics developers: Enable robots operating in dynamic environments to better remember past experiences and use external tools to adapt their reasoning.
Authors
Baojie Chen, Zijun Jia, Jing Zhong
Abstract
VLMs have shown promise for autonomous driving, yet still suffer from hallucination, weak spatio-temporal perception, and limited generalization. Recent methods improve reasoning and decision-making through CoT explanations, retrieval-augmented generation or the static injection of tool outputs. Although these mechanisms enrich the context, the model neither proactively perceives scene information nor accumulates experience after answering. To overcome these limitations, we present, to our knowledge, the first synergistic framework that tightly couples hierarchical memory with proactive tool invocation in a closed reasoning loop. Our contributions are threefold. (i) Hierarchical Driving Memory: a scene-level short-term memory maintains the dynamic scene state, and an evolving long-term memory retrieves reusable experience and tool strategies. (ii) Memory-Tool Synergistic Reasoning Framework: guided by the scene state and retrieved experience, the model adaptively invokes tools to refine its reasoning at inference time and consolidates reusable experience into a long-term memory pool offline. (iii) Data Generation and Two-stage Training Pipeline: verified memory-tool trajectories built by multi-step teacher rollout are used to train with SFT and GRPO. Our 7B model reaches an overall reasoning score of 80.03 and MCQ accuracy of 79.09% on DriveLMM-o1, surpassing the strongest baseline by 7.74 MCQ points and generalizes strongly across benchmarks. Notably, ablation and analysis studies validate the effectiveness of each component and further reveal the complementary roles of hierarchical memory. Short-term memory strengthens spatio-temporal understanding, improving STSBench accuracy by 24.2 points, while offline long-term memory consolidation yields an additional 3.57-point MCQ gain with all parameters frozen, demonstrating continual self-evolution through accumulated driving experience.