Conversational model predicts voice qualities from speech samples
Conversational Voice Aesthetic Model with Reinforcement Learning from Human Listeners
Computation and LanguageMachine LearningMultimediaSound
Summary
People often find it hard to describe qualities like emotion or delivery in voices because these aspects are personal and subjective. The researchers created a voice model called CVAM that listens to speech and writes descriptions about its sound and style. To train CVAM, they collected many opinions from human listeners and used a special method to make the model’s judgments match humans better. Their tests show CVAM agrees with people more closely than other existing speech models or even one human listener compared to others.
What this means in practice
- •For voice assistant developers: Automatically describe and rate voice qualities in conversational AI responses to improve naturalness and user engagement.$Commercial implications: Enables commercial voice assistant products to better tune responses for desired voice styles that appeal to users by aligning with human perception.
- •For call center quality analysts: Evaluate recorded agent speech to identify voice attributes linked to customer satisfaction without needing extensive manual review.
Authors
Xilin Jiang, Shun Zhang, Tejas Jayashankar, Yinghao Aaron Li, Osama Hanna
Abstract
We introduce Conversational Voice Aesthetic Model, a speech large language model for describing the voice aesthetics of real or synthetic speech responses in natural conversational contexts. Given a context and a response speech, CVAM describes salient moments that characterize the voice and predicts nine categorical attributes spanning gender, pitch, pacing, emotion, and delivery. The key challenge lies in perceptual fields such as emotion and delivery, which are inherently subjective and lack definitive ground truth. Therefore, we collect ~10 human annotations for each of 3k real and synthetic responses derived from the CANDOR corpus. CVAM is supervised finetuned on synthesized aesthetic descriptions and labels, then optimized with Group Relative Policy Optimization on human judgments. Experiments show that CVAM better agrees with human listeners than Gemini 3.1 Pro and open-source speech LLMs, and outperforms single-human-vs.-rest agreement. Together, we demonstrate the importance of grounding voice aesthetics in human perception and propose a principled framework for human alignment.