System turns experiment videos into interactive virtual labs
Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments
Multimedia
Summary
Watching science experiment videos shows you what happens, but you can't interact or try it yourself. The authors created Lab2Sim, a set of experiment videos with detailed instructions to help build virtual versions you can interact with. They also developed EduVideo2Code, a tool that breaks down video steps and turns them into interactive web pages you can run and test. Their tool improved the accuracy of reproducing the experiment setups compared to existing methods.
What this means in practice
- •For educational software developers: Create interactive virtual lab exercises directly from instructional science videos to enhance online learning platforms.$Commercial implications: Enables development of realistic virtual lab software products that simulate real-world experiments with validated setups for education providers.
- •For science museum exhibit designers: Develop interactive digital kiosks from demonstration videos to engage visitors with virtual hands-on science experiments.
Authors
Xusen Hei, Jiali Chen, Jinyu Yang, Yi Cai
Abstract
Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.