Geometry of agent memory weakly predicts answer correctness before generation
The Geometry of Agent Memory: Forecasting Retrieval-Bound Memory Failures, and Characterizing Its Limits
Computation and LanguageInformation Retrieval
Summary
This paper looks at whether the positions of conversation parts, memories, and questions in a shared space can help predict if an AI will answer correctly before actually trying. The authors tested different memory types and found only weak predictions, with their geometric method often no better than just knowing the question difficulty or AI system. They also show that similarity between memories and questions matters a bit but isn’t enough to reliably find errors. Their work helps us understand the limits of using geometry in AI memory but does not provide a dependable way to tell if an AI will answer right.
What this means in practice
- •For ai system developers: Identify when proximity-based memory retrieval might improve chatbot answer relevance using embedding similarity features.
- •For dialog system engineers: Use geometric features to select more relevant conversation memories for generating responses, potentially increasing answer accuracy.
Authors
Iliya Zhechev, Radoslav Ralev, Chris Coleman, Srijith Rajamohan, Jen Agarwal, Tyler Hutcherson, Simba Khadder
Abstract
We study whether geometric relationships between conversation turns, memory texts, and queries can predict answer correctness before generation. We evaluate four memory implementations with dense or hybrid retrieval and one observation-summary comparator. We embed these texts in a shared space. A logistic classifier uses 25 features describing query proximity, memory-conversation correspondence, and conversation structure. On LongMemEval-small, pooled AUROC is 0.640 over 2,367 question-system pairs, below a question-difficulty baseline of 0.655. Performance varies by question type: geometry reaches 0.841 on single-session-assistant questions, but system identity alone reaches 0.826. After controlling for difficulty and identity, geometry adds no statistically established pooled improvement (delta = +0.002, 95% CI [-0.010, 0.014]). A fixed-store context-selection experiment finds higher accuracy with proximity-ranked memories than with random subsets, consistent with ordinary retrieval relevance. On LoCoMo, nearest-turn similarity has modest predictive value for single-hop questions and near-chance estimates for several other categories. Training uses correctness labels; prediction uses pre-answer features and no additional LLM generation call. The results characterize associations between memory representations and correctness, rather than establish a reliable deployment diagnostic. In particular, embedding proximity does not verify fact coverage or distinguish memory-construction, retrieval, and generation errors.