Ai system converts tutoring videos into searchable learning transcripts

Towards Breaking the Learning System Wall Using Multimodal Tutoring Transcriptions

Human-Computer InteractionArtificial Intelligence

Summary

Understanding how students learn across different online platforms is hard because each system records data differently. The authors introduce a new AI tool that turns recorded tutoring sessions, including screen videos and conversations, into a single transcript showing both spoken words and student actions. This combined transcript can then link up with data from specific learning software, helping to map out what the student is learning. The approach aims to overcome current limits in comparing student learning across multiple platforms. Challenges remain, but these first steps may make it easier to study learning in more natural settings.

What this means in practice

  • For online learning platform developers: Use AI to create unified transcripts combining screen actions and tutor-student dialogue to better analyze and generalize student learning data.
  • For edtech product teams: Integrate multimodal transcription to enhance the monitoring and assessment capabilities of remote tutoring software platforms.$Commercial implications: Enables new features in tutoring products that offer detailed combined analysis of video and learning logs for sale to educational institutions.

A position paper. It proposes an approach and reports no results.

Authors

Danielle R. Thomas, Marie Cynthia Abijuru Kamikazi, Ashish Gurung, Ishan Miglani, Shivang Gupta, Zachary Levonian, Conrad Borchers, Kenneth R. Koedinger

Abstract

Past research using log data has faced the "learning system wall," whereby few methods exist for generalizing models of student learning across platforms. Increasingly, online learning is captured by richer forms of data, including dialog and video, with new affordances. An example of this is remote tutoring programs, where human tutors support students who use learning systems while video conferencing. Toward better platform-general modeling of learning, we introduce an AI-driven multimodal transcription system that processes screen-recording videos into unified screenplay-style transcripts containing audio dialogue and annotated learning log actions. We describe a planned method for temporally aligning AI-generated multimodal transcripts with MATHia learning logs and for identifying and classifying student learning processes to align with MATHia logs. Lastly, we highlight challenges and potential solutions in capturing learning processes in one system, offering initial steps towards generalizing log data across diverse systems.
View PDFOpen arXiv