Structured ai tutors improve student reasoning more than voice interaction
When AI Tutors Speak: Evidence from a Randomized Field Experiment
Human-Computer Interaction
Summary
Students often use AI to get answers, but this can make them think less. The authors tested an AI tutor designed to follow course lessons and found it helped students improve their reasoning skills more than just using regular AI tools. They also tested if speaking to the tutor or typing mattered and found it made no difference in learning, though students liked talking more. This means how the AI teaches matters more than how students talk to it.
What this means in practice
- •For educational technology developers: Design AI tutoring systems that emphasize structured teaching aligned with course content rather than focusing on voice interfaces to improve student reasoning outcomes.$Commercial implications: Supports building AI tutor products targeting education providers who want evidence-backed improvements in student learning.
- •For online course platform teams: Choose between voice or text AI interactions based on cost and student preference since voice does not increase learning despite higher delivery costs.
Authors
Shihao Yang, Marshall Van Alstyne, Chrysanthos Dellarocas
Abstract
Students increasingly study alongside generative artificial intelligence (AI), yet unguided access to fluent answers invites cognitive offloading, and there is little evidence on which configurations of AI tutoring produce learning. Two design margins are usually bundled together: pedagogical structure (how the tutor teaches) and interaction modality (how students talk to it). We separate them. In a preregistered randomized field experiment in a graduate corporate-finance course of an online MBA, we randomized 86 students between a structured tutor grounded in the course materials and a holdout in which consumer AI remained freely available. Within the tutored arm, each student's channel alternated weekly between voice and text, so the modality effect is identified within student. Structure mattered: tutored students gained 6.63 points more than ability-matched peers (p=.007), and the gain was concentrated in written reasoning, where the share of answers reaching relational quality rose from 8% to 49% in the tutored arm against 8% to 27% in the holdout. Modality did not matter for learning. The instructor's own final, on file for all 86 randomized students, shows the same direction (2.6 points of 100, with no difference on a pre-treatment midterm). Voice nearly doubled conversational interaction and cost 2.8 times as much to deliver, yet it produced weekly mastery statistically equivalent to text, even as students came to prefer it. Pedagogical structure shapes what students practice, and modality shapes how they interact with the tutor. Making an AI more humanlike does not by itself make it more educational.