Robots perform complex guqin music with precise hand coordination
Phrase-Level Robotic Guqin Performance: Bimanual Motion Planning and Audio-Tactile Interaction Monitoring
Robotics
Summary
Playing musical instruments with robots is hard because it requires exact finger movements and timing. The authors built a two-armed robot that can play the guqin, a traditional stringed instrument, by carefully planning each finger’s motion and coordinating both hands. Their system uses cameras, touch sensors, and sound feedback to adjust and monitor the robot’s playing. They tested it on a short tune with many notes and showed the robot could play the right notes with high accuracy.
What this means in practice
- •For robotics engineers: Develop robots that can perform delicate, coordinated bimanual tasks involving contact timing and acoustic feedback control.
- •For music technology developers: Use robotic systems with audio-tactile feedback to create automated performances of complex stringed instruments for research or entertainment.
Authors
Zhen Wang, Zhiheng Chen, Tianyuan Bao, Tianwei Zhang
Abstract
Recent advances in humanoid robotics and embodied intelligence have enabled robots to perform increasingly complex manipulation tasks. However, musical instrument performance remains a formidable benchmark, demanding not only collision-free trajectory execution but also precise contact timing, asymmetric bimanual coordination, and target acoustic outcomes on physical instruments. The guqin, a seven-string fretless zither, presents unique manipulation challenges due to its millimetric string spacing, transient right-hand plucking, and sustained left-hand harmonic contacts. In this work, we present a physical heterogeneous dual-arm robotic system for phrase-level autonomous guqin performance. We formulate guqin playing as a hybrid discrete--continuous execution problem and develop a hierarchical planning framework that coordinates working finger assignment, configuration continuity, obstacle avoidance, and tight bimanual contact schedules across consecutive musical events. The system integrates vision-guided instrument localization, tactile-based harmonic contact monitoring, and auditory feedback-informed plucking parameter calibration. Real-world experiments on a 25-event phrase demonstrate that the system reliably executes coordinated open-string and seventh-hui harmonic sequences on a physical guqin, achieving 93.6% and 96.8% event correctness across repeated trials.