PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise
2026-08-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created PREDACTBENCH, a new test to check how well dialogue AI systems work when their supporting tools give imperfect or noisy answers, focusing on education. They introduce new ways to measure how much users trust and rely on AI advice during multi-step conversations. Testing 13 advanced language models with real and synthetic education data, plus a human study, they found that current models often fail to help teachers correctly judge when to trust AI suggestions. Their benchmark aims to improve AI tools that assist teachers in decision-making with uncertain information.
Large Language Modelstask-oriented dialogue systemsnoisy toolstrust calibrationmulti-turn dialogueeducation datasetsAI-assisted decision-makingbenchmarkRelative AI-Reliancehuman-AI interaction
Authors
Abdulrahman AlRabah, Xiaocheng Yang, Dilek Hakkani-Tür, Abdussalam Alawini
Abstract
Large Language Models (LLMs) are increasingly deployed in task-oriented dialogue systems that support multi-step decision-making in high-stakes domains such as education, healthcare, and finance. However, existing benchmarks typically assume perfectly accurate tool outputs, overlooking the reality that deployed systems must operate with noisy tools and human decision-makers whose trust in the agent is itself uncertain. Such conditions are common in practice, for example, a clinician using a diagnostic prediction tool or an advisor relying on a model that forecasts student outcomes from historical records. We introduce PREDACTBENCH, a benchmark for evaluating dialogue agents paired with statistically imperfect tools, using education as a measurable testbed where ground truth outcomes and clear intervention decisions are available. First, we build a benchmark for AI-assisted human decision-making, where the AI uses noisy predictors to help guide a user. Second, we introduce episode-level Relative AI-Reliance (RAIR) and Relative self-reliance (RSR) metrics, extending prior trust calibration framework to multi-turn dialogue. Third, we evaluate 13 state-of-the-art closed and open source LLMs on two educational datasets, OULAD (real assessment trajectories from the UK Open University) and PREDACT-CS (60 courses with real final grade outcomes and synthetically generated weekly score trajectories), alongside a human study with instructors and teaching assistants. We find that when tools are noisy, SOTA models are supposed to provide visibility to teachers so that they do not over-rely on wrong suggestions or hallucinations, but current models fail to do that. We offer PREDACTBENCH to help build better LLMs as AI decision support systems to help teachers.