Teachers’ custom chatbots often fail in real use but can improve
EduWild: How Teacher-Configured Chatbots Fail in Real-World Use and How to Fix Them
Computation and Language
Summary
Many teachers make chatbots to help students learn, but these chatbots often mess up when actually used by learners. The authors collected a large set of real chats with teacher-made bots and identified common problems like skipping important steps or confusing students. They created a way to catch and fix the biggest issues using better instructions to the bots. When tested, this improvement made the chatbots work much better during real student conversations.
What this means in practice
- •For educational technology developers: Improve chatbot reliability by implementing failure-targeted prompts based on common teacher-configured issues.
- •For online learning platform teams: Deploy better-configured chatbots that reduce confusion and workflow mistakes for students interacting with automated tutors.
Authors
Ziqi Liu, Ruiwei Xiao, Yuxuan Li, John Stamper
Abstract
Teachers increasingly configure educational chatbots through natural-language instructions, but how these bots fail in real-world use remains poorly understood. We introduce EduWild, a dataset of 1,726 conversations with 217 teacher-configured chatbots, each paired with the instructions and configuration behind it, including 900 conversations from the intended end users of 35 deployed chatbots. We develop a 29-code taxonomy covering factual, workflow, diagnostic, and pedagogical failures and apply it using a human-calibrated LLM judge. The most frequent failures are lapses in following teacher intent --- bots skip required workflow steps (53.8% of end-user conversations), omit expectation setup (33.4%), and overload learners with information or questions (37.1%). Guided by these findings, we design a prompt-based harness targeting the six most frequent failure modes. In a study with 200 college students across ten bots, the harness reduces the mean number of target failure codes per conversation from 1.406 to 0.489, a 65.2% relative reduction with bots weighted equally. These findings show how real-world failure analysis can guide practical improvements to educational chatbots.