Persistent AI agents evaluated for identity recall and behavior
Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents
Artificial IntelligenceMultiagent Systems
Summary
Chatbots and AI agents often need to remember who they are and act consistently over time. This paper presents PAI-Bench, a new test that checks not only if agents can recall identity facts but also if they express and act on them correctly. The authors tested different AI profiles and found that simple reminders helped agents show more consistent identity features. They also noticed differences between AI systems in how they respond to identity tasks. PAI-Bench offers a way to measure different parts of an AI’s identity reliably.
What this means in practice
- •For ai developers: Test and improve how deployed AI agents remember and express their identities under changing instructions and updates.
- •For customer service teams: Check deployed AI assistants for consistent personal identity signals that affect user trust and satisfaction.
Authors
Zhenyu Zhao, Roy Zhao
Abstract
Persistent agents need evaluations that distinguish identity facts they can recall from those they express and enact. We introduce PAI-Bench, a provider-neutral benchmark for fidelity to a versioned, update-governed identity contract. It separates recall, composition, behavioral enactment, resistance, persistence, lineage, and role-conditioned updates while keeping scoring oracles outside the target process. Two frozen campaigns cover sixteen synthetic profiles, thirty-two probes, and three independently initialized target configurations, yielding 1,536 retained responses. A judge-independent literal audit finds direct-parent identifiers in 48/48 atomic responses but only 1/48 implicit self-portraits. On eight profiles, explicit field cues increase joint presence of three identity identifiers from 0/8 to 7/8 under the same four-sentence instruction. A separate startup body-label substitution increases full-designation presence from 1/8 to 7/8 while parents remain absent. These contrasts reveal prompt-dependent component selection and component-specific sensitivity to startup cues in the tested deployments. Replaying identical factorial responses also yields a Claude headline mean 12.5 percentage points below Astra's, demonstrating evaluator sensitivity separately from target behavior. The studies use single target samples per condition, with post-hoc audits and follow-ups. PAI-Bench provides a reproducible evaluation protocol for measuring factual availability, identity expression, and behavioral enactment as distinct aspects of identity-contract fidelity.