CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
2026-06-01 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created CRAB-Bench and RUSE to better test how well language model agents handle realistic, complicated tasks with tricky wrong options and human-like user behaviors. CRAB-Bench makes tasks that need careful thinking because only a few possible answers are correct among many distracting ones. RUSE simulates users based on real human behaviors instead of simple scripted responses, making interactions more lifelike. They found that even top models struggle with these tests, especially when realistic user behaviors are involved, and that some users’ ways of sharing information make it hardest for agents to solve tasks correctly.
LLM agentsConstraint graphUser simulationPass@1Task dependenciesDistractorsBehavioral dimensionsInformation disclosureImplicit correctionsPersona modeling
Authors
Danqing Wang, Akshay Sivaraman, Lei Li
Abstract
Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.