From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents
2026-08-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors point out that current tests for personalized AI agents are too simple because they don't consider how people’s preferences change over time or how agents should act based on those preferences. They introduce IBA-Bench, a new test that uses real, messy interaction histories to see if an AI can figure out and follow user preferences without being directly told. They also create IBA-Agent, a system designed to better handle conflicting preferences and improve how well the AI matches user behavior. Their experiments show that existing AI agents still struggle with true personalization, but their new agent makes noticeable improvements.
Large Language ModelsPersonalizationAutonomous AgentsUser PreferencesLongitudinal InteractionBehavioral AlignmentImplicit FeedbackBenchmarkTrajectory-level AlignmentPreference-conditioned Task Execution
Authors
Jiajia Song, Bobo Li, Haiwen Yi, Zibo Ji, Meishan Zhang, Hao Fei, Min Zhang, Mong-Li Lee, Wynne Hsu
Abstract
Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful. Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles. Such designs fail to capture the complexity of evolving user preferences and neglect preference-conditioned task execution-a discrepancy we term as the knowledge-to-action gap. To address this challenge, we introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories that contain noise, implicit cues, and temporal inconsistencies. Unlike prior work, IBA-Bench evaluates whether an agent can execute tasks while satisfying implicit user constraints inferred from historical interactions. We further propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment. Experiment results on IBA-Bench show that effective personalization remains a significant challenge for state-of-the-art LLM agents, and the proposed IBA-Agent substantially improves behavioral alignment in complex scenarios across nine application domains.