Summary
This paper studies when storing past experiences helps AI systems that control driving in a loop. The researchers found that apparent improvements often come from the makeup of the initial data pool and the specific tokens added to prompts, rather than the memory design itself. They created a carefully controlled test environment to separate these effects and showed that some reported benefits disappear under stricter testing. Their work warns that claims of memory improvements should be treated cautiously and offers tools to better evaluate AI driving agents.
What this means in practice
- •For autonomous vehicle developers: Use controlled benchmarks and token-matched fillers to rigorously evaluate memory modules in LLM-based driving agents.
- •For ai prompt engineers: Design prompts with caution tokens that match memory-related token counts to better assess memory impact on LLM agent behavior.
Tested on simulated data.
Abstract
When does experience memory help a closed-loop LLM driving agent? Published gains cannot say, because the two factors we find to matter most, which warmup experiences the pool contains and what any injected tokens do to the prompt, are controlled in none of them. We answer with a controlled evaluation on one audited HighwayEnv testbed (1Hz, gpt-4o-mini for the decisive contrasts). We treat the warmup pool as a resampled random factor (ten disjoint pools) and add token-matched irrelevant-filler controls, oracle-certified diagnostics, caution-pricing metrics, and quantified audits of the baseline ports. The probe suite spans flat-text retrieval (a DiLu port), a single-frame grid port (RESPOND-inspired), our episodic temporal-graph memory, a retrieval-free consolidated rulebook, and a flat K-step observation-history baseline. Four findings follow, each a property of this testbed. (1) Headline memory gains belong to the warmup draw. The reported pool is the best of ten for both retrieval memories, which beat No-Memory in only one pool of ten (sign test p=0.011), and even the rulebook's apparent pool-immunity collapsed on five fresh draws. (2) No retrieval design tested exceeds token-matched filler. The collision-only merge ceiling is reached by a single 15-token caution instruction, and metrics that price caution erase the rulebook's lead. (3) Representational sophistication is nowhere robustly operative. (4) Baseline port quality silently inverts conclusions. On this testbed, a single-pool comparison without filler controls would have reported memory gains that these controls reverse. We release the testbed, audited ports, seeds, and caches.