VR system uses AI to help teach stage lighting more effectively
LumiNote: LLM-Assisted Multimodal Instruction for VR Stage Lighting Education
Human-Computer Interaction
Summary
Teaching stage lighting is hard because it involves abstract ideas and complicated equipment. The authors created LumiNote, a virtual reality tool that uses an AI language model to turn an instructor’s spoken instructions into clear visual notes, demonstrations, and explanations inside VR. They found it helped instructors explain hard-to-describe concepts, but experts still needed to guide the system for detailed equipment or positioning. The tool shifted teachers’ effort from setup to teaching, although some explanations were tricky for beginners to understand.
What this means in practice
- •For vr training developers: Build interactive VR lessons that convert instructor speech into visual and practical stage lighting training aids.
- •For lighting technicians: Use VR-based AI tools for hands-on practice and clearer instruction when learning complex lighting setups.
Authors
Danxuan Liang, Chun Yin Li, Zheng Wei, Xian Xu, Meng Xia, Huamin Qu, Wai Tong
Abstract
Stage lighting education requires instructors to bridge abstract concepts, technical operations, and learner-understandable representations. While Virtual Reality (VR) removes physical constraints, existing systems provide limited support for live instruction. We present LumiNote, an LLM-assisted VR system that transforms spoken pedagogical intent into instructor-reviewable spatial annotations, executable demonstrations, and linguistic support. In an exploratory study with 3 instructors and 24 students, we examined how instructors incorporated LumiNote into familiar lighting topics and how students received the resulting representations. We found LLM assistance most valuable for expressive, under-specified goals, but requiring greater expert intervention for fixture-specific or spatial configuration requests. Instructors engaged with generated suggestions as a controllable refinement process, shifting effort from manual setup toward pedagogical expression. However, representations that externalized expert reasoning did not always align with novice comprehension. These findings characterize LLM-assisted VR instruction as a domain-grounded mediation process among expert expression, executable operations, and learner-facing representations.