TokenMapper enables direct speech token translation between models
TokenMapper: A Step Toward Interoperable Speech Token Translation
Machine LearningArtificial IntelligenceSound
Summary
Different speech processing systems break down voice into sequences of tokens, but these tokens don’t match up between models, so communication between systems is slow and can lose information. The authors created TokenMapper, a tool that translates these tokens directly between speech models without turning them back into sound first. This method reduces delays and improves accuracy compared to traditional ways. Their tests show the translations are close to the original quality while being faster.
What this means in practice
- •For voice assistant developers: Enable voice agents to transfer speech data between different speech models quickly without converting back to audio, reducing latency and maintaining quality.
- •For speech translation engineers: Improve speech-to-speech translation systems by directly mapping tokens between heterogeneous speech tokenizers, speeding up processing and preserving speech details.
Authors
Tal Kozakov, Tal Rosenwein, Eliya Nachmani
Abstract
Neural audio codecs discretize speech into token sequences, but the resulting token spaces differ in vocabulary and codebook structure, preventing direct communication across models. This limitation affects applications such as conversational voice agents and speech to speech translation systems where multiple speech models must interact. As a result, transferring information between speech systems typically requires decoding to waveform audio and re-encoding with a second tokenizer, increasing latency and introducing potential information loss. To address these limitations, we present TokenMapper, a direction aware framework for direct token to token translation between heterogeneous speech tokenizers in the discrete domain. TokenMapper supports structurally mismatched token spaces, including mappings between single codebook and multi codebook representations, under a shared effective token rate. Experiments on GLM-4-Voice, MiMi and DualCodec show consistent cross model performance. Specifically, translation WER approaches native reconstructions within 2.5-6.8% absolute WER, human MOS for TokenMapper outputs ranges from 2.29 to 4.39, following the same direction level trends as UTMOS and end to end latency is reduced by 4.8-94.5% relative to waveform bridging, reaching up to 972 ms per utterance. These results provide a practical step toward cross model speech token interoperability without intermediate waveform reconstruction.