Ai teaching assistants with guidance style affect programming learning
Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming
Human-Computer InteractionArtificial Intelligence
Summary
Helping students learn programming using AI assistants can work in different ways. The authors studied two types of help: one that asks questions to guide students and another that gives direct instructions. They also looked at whether the AI understands the student's current work or not. The study found that students disliked the question-asking AI when it fully knew their work, sometimes avoiding it and doing worse. So, being strict or too aware of context doesn’t always help students learn better.
What this means in practice
- •For educational software developers: Design AI tutors that balance teaching style and context to improve student satisfaction and learning outcomes in programming courses.
- •For online course platforms: Integrate AI teaching assistants tuned for context awareness and tailored guidance to reduce student reliance on unauthorized external help.
Authors
Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, Amanpreet Kapoor
Abstract
AI teaching assistants (AI TAs) backed by large language models (LLMs) and pedagogical guardrails are increasingly being integrated into programming courses, providing students with scalable access to hints, conceptual explanations, and code-level feedback. However, guardrails may also create friction. If students feel that the support provided is overly restrictive or poorly contextualized to their current progress, they may bypass approved tools for general-purpose LLMs. To investigate how AI TA design affects students' learning experiences, we conducted a randomized controlled trial with 132 students in an introductory programming course. Students completed three tasks related to code-writing and debugging and were randomly assigned to one of four AI TAs varied across two dimensions: pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (no context vs. full context of the problem and student solution). We examined students' perceptions, interaction behaviors, and evidence of post-task comprehension. Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion. Descriptively, this condition also showed the highest observed interaction stress, the highest rate of external LLM use, and the lowest proportion of post-task explanations demonstrating full comprehension, though these differences were not statistically significant. These findings suggest that guardrailed AI TAs are not automatically better for learning. Instead, their effectiveness depends on how pedagogical guidance and contextual awareness are balanced in ways that students experience as useful, supportive, and worth continuing to use.