Generative AI helps improve learning but not critical thinking skills
Critical Thinking with Generative AI: A Constraint-First Design Pilot of a Thinking-Partner Intervention
Human-Computer Interaction
Summary
Generative AI tools like ChatGPT are used in classrooms fast, but it's unclear if they help students think critically. The authors tested using ChatGPT as a thinking partner in a college statistics course and found students learned more about statistics and AI use, but their standard critical thinking scores did not improve. Students mostly used the AI to check answers instead of having deep discussions with it. The study suggests how future teaching with AI might be designed to better support critical thinking.
What this means in practice
- •For education technology developers: Design AI-based learning tools that improve domain knowledge and AI literacy while recognizing limitations in boosting critical thinking.
- •For curriculum designers: Create course materials that guide students to engage with AI tools beyond validation to encourage deeper conceptual understanding and dialogue.
Authors
Fatima Tuz Zahra, Jiangen He, David M. Bowers, Wei Wang
Abstract
Generative AI (GenAI) tools entered higher education classrooms faster than the field was able to study their effects on learning. One concern is that GenAI may displace the critical thinking and AI literacy that students will need after graduation. This paper reports a Design-Based Research pilot of a GenAI-assisted critical thinking framework, in which ChatGPT was used as a thinking partner in an undergraduate research methods and statistics course during Spring 2025 (N = 14). The mixed-methods design combined pre- and post-intervention measures of statistical learning (AASCDM), AI literacy (MAILS), and critical thinking (WGCTA) with instructor field notes, student artifacts, and student-AI interaction logs. Pre-post tests showed gains on every AASCDM dimension and on eight of nine MAILS dimensions, while WGCTA percentiles did not change. Qualitative analysis identified four themes: the ways students positioned the LLM (as answer generator, validator, or co-thinker); the depth of student engagement (procedural vs. conceptual); occasional humanizing of the tool; and the role of curriculum design in shaping each of the prior three. Read together, the findings indicate that one semester of GenAI-assisted instruction can move domain learning and self-reported AI literacy but does not move standardized critical thinking, and that the modal student-LLM relationship is one of validation instead of dialogue. We end with design principles for the next iteration of the framework and implications for research on adaptive and personalized learning.