HeyGen Voice clones voices with matching scene sounds
HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos
Sound
Summary
Many voice cloning technologies only copy how someone sounds in the original recording, so if the recording was made on a phone, the cloned voice sounds like it's on a phone everywhere. The authors created HeyGen Voice, which keeps a person's natural voice but changes how it sounds depending on the scene shown in a video, like a quiet room or a cathedral. It uses a special machine learning model that learns from voices, pictures, and captions about the scene to make the speech sound more realistic. Listeners preferred this approach over other commercial voice cloning tools.
What this means in practice
- •For video content creators: Generate talking videos where the speaker’s voice matches the visual scene’s acoustics and environment.$Commercial implications: Enables commercial production of realistic digital avatars that sound consistent with video scenes, improving audience immersion.
- •For digital assistant developers: Create voice assistants whose speech delivery adapts to different visual contexts shown on devices or environments.
Authors
Pavel Andreev, Andrei Gusev, Adithya Narayan, Yi Ren, Ivan Somov, Zijian Li, Sergey Demyanov, Viktor Goriachko, Alan Tan, Nik Nolte, Sam Prokopchuk, Andy Pan, David Chou, Kirill Leonov, Penghan Wang, Jiewei Yuan, Zhibin Hong, Rong Yan
Abstract
Most voice-cloning systems are built for audio alone: they copy the reference recording's delivery and acoustics, so a voice cloned on a phone sounds like a phone recording whether the speaker is pictured in a studio, on a street, or in a cathedral. Speech for a talking video should sound like the pictured person speaking in the pictured place. We present HeyGen Voice, a scene-aware voice-cloning model that separates who is speaking from how and where: it keeps the speaker's voice from an everyday recording while the target image sets the delivery and acoustics. Prior image-conditioned TTS does half of this job: it either adds scene acoustics without keeping a specific speaker or infers a voice from a face. HeyGen Voice is a multimodal autoregressive transformer conditioned on the image, speaker and scene captions, and the voice recording, trained in five stages that move from multilingual TTS pretraining through simulated and real cross-scene adaptation to reinforcement and preference optimization. Two clone policies trade scene adaptation against fidelity to the original recording, and an audio anchor holds long scripts in one voice and one acoustic environment. Listeners prefer HeyGen Voice over leading commercial systems across a third-party arena, production votes, and talking-video studies: it ranks first (Elo 1201) on the public Artificial Analysis controlled-voice TTS leaderboard (October 2026); in a blind test, users cloning their own voice chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes; and volunteers rated talking videos of their own digital twin about 10 to 30 points higher (out of 100) when voiced by HeyGen Voice than by leading commercial alternatives. Try HeyGen Voice and hear samples at https://heygen.com/heygen-voice.