VidTutorAssistant automates answering programming tutorial questions accurately
VidTutorAssistant: Automating Responses to Programming Tutorial Questions
Software Engineering
Summary
Video tutorials on programming often have many questions in their comments that creators cannot answer quickly. The authors made VidTutorAssistant, a web tool that reads a video's transcript and uses AI to find and answer viewer questions based on what was actually said in the video. Their tests show it can identify programming languages and questions with high accuracy and produce answers that are more correct and complete than the original video creators’ replies. This means learners can get better help from video tutorials without waiting for a human to respond.
What this means in practice
- •For online learning platforms: Automatically provide accurate answers to user questions on programming tutorial videos using extracted video transcripts and AI.$Commercial implications: Enables platforms to offer enhanced learner support and engagement by integrating automated AI replies grounded in tutorial content.
- •For video content creators: Reduce workload by automating responses to programming questions in video comments, improving viewer engagement without manual replies.
Authors
Ahmad Tayeb, Sonia Haiduc, Mohammad D. Alahmadi
Abstract
Programming tutorial videos on YouTube are an important information resource for software developers and students, and their comment sections have evolved into active spaces where viewers ask follow-up questions. The volume of these questions, however, often exceeds what content creators can address, leaving learners without the clarifications they need. We present VidTutorAssistant, a web platform that automates responses to viewer questions on programming video tutorials. VidTutorAssistant implements a retrieval-augmented generation pipeline that extracts a video's transcript, then segments it and embeds it. It then classifies each viewer comment as being a question or non-question, retrieves the most relevant transcript segments to each identified question via cosine similarity, and then generates an answer to the question using an LLM (GPT-4), while grounding the response using the retrieved transcript segments as context. We validate VidTutorAssistant through a study on a subset of 440 user comments selected from a larger dataset of 105,553 comments extracted from 7,522 Python and Java tutorials. VidTutorAssistant is evaluated on various criteria: a) its ability to identify the programming language in a video, achieving a 0.99 accuracy; b) its ability to classify comments into questions and non-questions, reaching a 0.96 accuracy; and c) its ability to produce correct and complete answers to questions, producing 98% correct and 99.5% complete responses, compared with 89% and 90% for the original creators' answers.