Pace-mem improves multimodal agents question answering with smarter memory retrieval
PACE-Mem: Prefix-Aware Complementary Evidence for Multimodal Agents
Computer Vision and Pattern Recognition
Summary
Multimodal agents remember many similar events, which can make it hard to find the best information to answer questions accurately. The authors propose PACE-Mem, a method that picks the right amount of helpful memories without repeating too much overlapping information. It does this by scoring and selecting evidence in a way that balances relevance and uniqueness, deciding exactly how much to include for each question. This approach works without changing how memories are stored or how answers are generated, and it improves accuracy on long video tasks.
What this means in practice
- •For multimodal system developers: Enhance long-term memory retrieval in multimodal agents to improve question-answering accuracy without retraining their memory or response models.
- •For video analytics teams: Improve the selection of relevant video segments from long recordings when responding to queries, reducing redundant information.
Authors
Shidu Ren, Rong-cheng Tu, Hang Zhou, Xiao Luo
Abstract
As multimodal agents accumulate experience, their long-term memories contain many similar observations of recurring scenes, actions, and entities. Answer quality depends on the evidence retrieved from these memories into the model's context. However, query-relevant memories may contain overlapping information, and a fixed retrieval depth may return too little or too much evidence for a given question. To address these challenges, we propose PACE-Mem, Prefix-Aware Complementary Evidence, which constructs a complementary evidence sequence and determines its length from the same selection trajectory. For structured memories, query-guided pooling combines records within each clip to obtain relevance scores and representations. Using these relevance scores and representations, maximal marginal relevance (MMR) selection balances query relevance against overlap with previously selected evidence. From the resulting score trajectory, a prefix-length selection algorithm determines retrieval depth by comparing length-normalized adjacent declines, accounting for both score changes and the amount of evidence accumulated. PACE-Mem is a training-free, plug-in retrieval method for clip-based agent memory systems, leaving memory construction and answer generation unchanged. Across three long-video benchmarks and two agent frameworks, PACE-Mem improves question-answering accuracy by an average of 1.6% over Adaptive-k, the strongest baseline at comparable mean retrieval depths. Further analysis reveals that jointly adding and removing evidence is associated with larger net accuracy gains than either change alone.