Text to music systems rated by accuracy and speed for fair comparison
TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking
SoundMachine LearningMultimedia
Summary
Creating music from text descriptions is getting more common, but it’s hard to compare different systems because they work in different ways and use different measurements. The authors made a system called TTM-Bench that tests how well these music-generating tools match the text descriptions and how fast they work. They found that better matching doesn’t always mean faster performance, so it’s important to measure these qualities separately. This tool helps people understand and compare text-to-music generators more fairly and clearly.
What this means in practice
- •For audio software developers: Compare and improve text-to-music generation algorithms on both quality and speed using a unified testing method.
- •For cloud music service providers: Assess hosted text-to-music services on cost and performance to select or tune offerings for customers.$Commercial implications: Enables pricing and selection of text-to-music cloud services based on standardized quality and efficiency measures.
Authors
Giorgia Adorni, Michela Papandrea, Battista Rimoldi, Tiziano Leidi
Abstract
Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essential, yet reliable performance comparison remains challenging. This difficulty stems from differences in system architecture, supported conditioning information, and access mode, as well as heterogeneous and fragmented metrics that cannot be applied uniformly across systems. To address these challenges, we introduce TTM-Bench, a framework that defines a common protocol for systematic, reproducible performance benchmarking of contemporary TTM systems. It evaluates performance along two dimensions: musical-content alignment, quantified by interpretable semantic, genre, and musical-descriptor agreement scores against a common musical specification and summarized by an aggregate score; and computational efficiency, characterized by generation latency and real-time factor, alongside resource use for local models and cost for hosted services. We demonstrate the framework through a preliminary comparative case study, illustrating the complementary evidence captured by these dimensions. The results show that higher musical-content alignment does not systematically coincide with lower computational demands, highlighting the importance of assessing TTM performance through distinct, interpretable measures rather than a reductive overall indicator.