Multimodal system helps edit concert videos from multiple cameras

Multimodal Concert Video Editing: System Design and Exploratory Evaluation

MultimediaComputer Vision and Pattern RecognitionSound

Summary

Editing videos of multi-camera concerts is tricky because editors must choose the best camera view at the right time, balancing what instruments sound and look most interesting. The authors created a system that scores different camera shots by combining information about which instruments play, which are visible, how clear the image is, and the music beat. They tested this system with experts on a four-camera concert and found mixed results, suggesting that future work should include more context and consider longer sequences of shots. Their system design can help researchers test these improvements and understand how different parts affect video quality.

What this means in practice

  • For video production teams: Automatically choose camera angles in multi-camera concert footage using combined audio and visual cues to streamline editing.
  • For live event streamers: Enhance live concert streaming by selecting camera shots based on instrument activity and visual quality in real time.

Tested on one dataset.

Authors

Lennart Schiweck, Adrian Kempf, Michelle Retig, Moritz Feucht, Tim Mangliers, Benjamin Himpel

Abstract

Editing multi-camera concerts requires coordinated decisions about musical relevance, visual quality and cut timing. We present a modular framework that combines instrument activity from multitrack audio, visible instruments and image sharpness with beat-based timing and probabilistic camera selection. An interpretable scoring model links these components to the choice of views over complete shot intervals. Component tests and an exploratory expert study on a four-camera concert yield mixed assessments across the editing variants. The findings motivate extending local audiovisual relevance with contextual and sequence-level criteria. The architecture provides a basis for testing these extensions and studying how component behaviour affects the quality of complete edits.
View PDFOpen arXiv