HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents
2026-06-15 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors address a problem in long-term learning systems where it's hard to know if updating memory helped or hurt, since later errors can confuse the cause. They propose HiMPO, a method that better judges the value of memory updates by comparing what useful information is gained before and after the update. HiMPO then filters feedback to give fair credit only when the update truly contributed to success. Their tests show this approach improves the system's learning and avoids blaming memory updates for unrelated mistakes.
long-horizon agentsmemory mechanismscredit assignmentreinforcement learninghindsightmemory policy optimizationcausal entanglementopen-domain taskscompressive memoryattribution fidelity
Authors
Jiangze Yan, Yi Shen, Wenjing Zhang, Jieyun Huang, Zhaoxiang Liu, Ning Wang, Kai Wang, Shiguo Lian
Abstract
Long-horizon agents rely on memory mechanisms to compress interaction history, but optimizing memory writing faces a distinct credit assignment challenge: a memory update may be rewarded or penalized due to downstream tool failures, noisy observations, or reasoning errors rather than its own contribution. This causally entangled credit can lead agents to discard useful evidence or preserve irrelevant information. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework for assigning less-entangled credit to memory-writing actions in long-horizon agents. HiMPO first estimates the local utility of a memory update by comparing the task-relevant information recoverable from the previous and updated memories under the same pre-write state. It then uses hindsight relevance as a bounded retrospective filter that attenuates memory credit when local utility is not supported by the target outcome. The resulting memory-specific advantage is applied only to memory tokens, while trajectory-level rewards optimize the rest of the agent behavior. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves over strong memory-based and RL-based baselines while preserving compressed-context efficiency. Controlled interventions further show that HiMPO reduces blame leakage from tool-induced errors and improves attribution fidelity of memory updates.