Memory repair in language models often costs more than re-reading evidence
When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents
Computation and Language
Summary
This paper looks at how language models update their facts when the original supporting information changes. The authors compare two strategies: fixing memory where changes happen, or re-reading the updated documents. They find that fixing memory locally uses fewer words internally but usually costs more overall than re-reading, especially as records get longer. This suggests when evidence changes, it might be cheaper to just re-check the documents than try to repair stored memories.
What this means in practice
- •For medical data teams: Manage updates in patient records by deciding when to revise stored information or re-check original documents to optimize processing costs.
- •For legal document analysts: Improve accuracy and cost efficiency when handling changes in supporting evidence within large document sets by balancing memory repair and re-reading.
Authors
Wenhui Chu
Abstract
When documents supporting an agent's derived facts are revoked or replaced, should it repair memory or re-read current evidence? We introduce an evidence-revision evaluation on medication- and problem-list tasks from public ICU records. Under revocation, replacement and control events, we compare full and source-filtered re-reading with caching, rebuilding and graph-local repair across two 7B models. Memory is supplied in full without retrieval, and costs include ingest, revision and every use. On short records, local repair uses 5-10$\times$ fewer revision tokens than rebuilding, yet every memory pipeline costs at least twice full re-reading in held-out conditions. In a small pre-specified development sweep, adding task-ineligible documents extended records to about 10,000 tokens; at that length, memory's mean cumulative cost fell below full re-reading's after 2-14 uses, partly through truncated extraction, while source-filtered re-reading remained cheapest. In the replacement study, none of the four primary confirmatory tests reached statistical significance. These results show why the cost of agent memory after evidence revision must be assessed against source-filtered re-reading over the full pipeline.