Ai teaching partner boosts confidence but not critical skills learning
Fluency Without Evidence: Constraint-First Design and the Limits of Self-Report in AI-Assisted Learning
Human-Computer Interaction
Summary
This paper tested a special AI tool designed to help students think more deeply by asking them to explain their answers before the AI responded. The researchers found that after using this tool, students felt they understood AI better, but they did not actually improve in critical thinking, awareness, or knowledge. The improvements were mostly in what students reported themselves, which might just mean they felt more confident rather than learned more. The study also showed that records of how students interacted with the AI were incomplete, suggesting that self-reports alone might not reliably show real learning with AI tools.
What this means in practice
- •For educational technology developers: Design AI teaching tools that collect detailed interaction data to better assess student learning beyond self-reports.
- •For online course platforms: Implement AI partners with constraint-first designs to improve user confidence in AI concepts without assuming gains in critical thinking.
Authors
Fatima T. Zahra, Wei Wang, Frances Harper, Jiangen He
Abstract
A generative AI teaching partner should support reasoning over supplying conclusions; however, this has not been tested against learning in an authentic course. Drawing on design-based research, we specify the position as a conjecture map and report a first design cycle in two graduate-level research methods courses. Students used an AI teaching partner employing a constraint-first sequence requiring them to state and justify positions before receiving questions. Pre- and post-measures of AI literacy, critical thinking, and metacognitive awareness were collected alongside interaction records. AI literacy increased, concentrating in understanding AI, whereas critical thinking, awareness, and knowledge did not change. Since changes were limited to self-report measures, they may reflect growth in confidence instead of capacity. Interaction records, meanwhile, showed brief exchanges, uneven enactment of the constraint-first sequence, and missing records. These findings show why AI-supported learning requires interaction records to provide a more defensible basis for AI-supported designs than self-reports.