Personal AI memory improves rating predictions with correct histories
A Controlled Audit of Personal AI Memory for Rating Prediction
Machine Learning
Summary
The paper looks at whether personal AI systems rely more on a user's past item ratings or just their overall rating habits to predict new ratings. The authors scrambled the order of past ratings while keeping other details the same to test this. They found that having the correct history of item ratings helps the AI make better predictions, especially on one dataset called Coat, but the effect was less clear on another dataset, MovieLens. The study also shows that how AI systems use past data and how reliable their predictions are need to be evaluated carefully on their own.
What this means in practice
- •For recommendation system engineers: Improve rating prediction accuracy by verifying that AI models correctly use individual users' exact rating histories.
- •For data scientists in e-commerce: Evaluate how changes in user rating histories affect AI predictions to refine customer behavior models for personalized recommendations.
Authors
Shivam Gupta
Abstract
In structured rating prediction, does a personal AI use historical item-rating associations, or mainly the user's rating tendencies? We audit this distinction by permuting historical ratings within each user while preserving the exact rating distribution, item support, and metadata. We combine this control with full history, native memory extraction, and matched numerical readers in a publicly frozen evaluation of 400 held-out user profiles and 6,160 target ratings across Coat and MovieLens. On Coat, the tested Qwen-written Mem0 pipeline increases user-macro mean absolute error relative to full history by 0.084 for Qwen and 0.149 for Phi; both family-adjusted bootstrap intervals exclude zero. Correct historical assignments help both readers on Coat, but the corresponding MovieLens effects are smaller and inconclusive after adjustment. A history-only ridge reader outperforms Qwen in both domains and Phi on MovieLens, while the Coat Phi comparison is unresolved. All 2,800 reader calls, including 150 invalid outputs, are retained under a fixed fallback rule. A separate implementation verifies inputs, metrics, and all ten primary contrasts. The contribution is a reproducible diagnostic study showing why extraction, association use, output reliability, and reader choice require separate evaluation.