The Qwen team at Alibaba Cloud has announced the open-sourcing of Qwen3-TTS, a comprehensive family of multilingual text-to-speech (TTS) models. This new offering aims to address three primary objectives within a unified framework: high-quality speech generation, precise voice cloning, and flexible voice design.
A Diverse Model Portfolio for Enhanced Voice Control
Qwen3-TTS leverages a 12Hz speech tokenizer alongside two distinct language model sizes, 0.6 billion and 1.7 billion parameters, structured to support its core functionalities. The initial release comprises five models, catering to various user needs. The Qwen3-TTS-12Hz-0.6B-Base and Qwen3-TTS-12Hz-1.7B-Base models are engineered for general TTS applications and voice cloning tasks. For those seeking pre-defined vocal characteristics, the Qwen3-TTS-12Hz-0.6B-CustomVoice and Qwen3-TTS-12Hz-1.7B-CustomVoice variants offer access to promptable preset speakers. Additionally, the Qwen3-TTS-12Hz-1.7B-VoiceDesign model enables users to craft unique voices from descriptive natural language inputs, complemented by the Qwen3-TTS-Tokenizer-12Hz codec.
The entire suite supports a broad linguistic spectrum, encompassing Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, and Italian. CustomVoice models arrive with nine pre-selected timbres, each featuring a brief descriptor of its unique sound and speaking style, such as 'Vivian,' a bright Chinese female voice, or 'Ryan,' a dynamic English male voice. The VoiceDesign model allows for intricate voice creation, responding to directives like 'speak in a nervous teenage male voice with rising intonation,' which can then be combined with Base models using a short reference audio clip.
Advanced Architecture for Real-Time Streaming
Qwen3-TTS employs a dual-track language model architecture, where one track processes discrete acoustic tokens from text, while the other manages alignment and control signals. The system underwent extensive training on over 5 million hours of multilingual speech data across three stages, progressing from general mapping to high-quality data and finally to support for long contexts, up to 32,768 tokens.
Central to its performance is the Qwen3-TTS-Tokenizer-12Hz codec. Operating at 12.5 frames per second, equivalent to approximately 80 milliseconds per token, it utilizes 16 quantizers with a 2048-entry codebook. Performance metrics on the LibriSpeech test set demonstrate its superiority over several contemporary semantic tokenizers, including SpeechTokenizer and XCodec, achieving a PESQ wideband score of 3.21, STOI of 0.96, and UTMOS of 4.16, all while maintaining a comparable or lower frame rate.
The tokenizer's design as a pure left-context streaming decoder facilitates immediate waveform emission once sufficient tokens are processed. With four tokens per packet, each streaming packet delivers about 320 milliseconds of audio. An efficient decoder and BigVGAN-free architecture contribute to reduced decoding costs and streamlined batch processing. Early tests on a vLLM backend indicate impressive streaming latency, with first-packet delivery around 97-101 milliseconds for the 0.6B and 1.7B Base models at concurrency 1, and maintaining robust performance even at higher concurrency levels.
Precision Alignment and Multilingual Excellence
The system's post-training phase integrates a sophisticated alignment pipeline. This includes Direct Preference Optimization (DPO) to align generated speech with human preferences, followed by GSPO with rule-based rewards for improved stability and prosody. A final speaker fine-tuning stage on the Base model ensures target speaker variants preserve the core capabilities of the general model.
Instruction following is implemented via a ChatML-style format, where textual commands for style, emotion, or tempo are prepended to the input. This versatile interface underpins VoiceDesign, CustomVoice prompts, and detailed edits for cloned voices.
Benchmark evaluations confirm Qwen3-TTS's strong capabilities. On the Seed-TTS test set, the 1.7B-Base model achieved a Word Error Rate (WER) of 1.24 on English, positioning it as state-of-the-art among compared systems for zero-shot voice cloning. In multilingual TTS evaluations across ten languages, Qwen3-TTS secured the lowest WER in six languages—Chinese, English, Italian, French, Korean, and Russian—while showing competitive results in the remaining four. Crucially, it recorded the highest speaker similarity across all ten languages when benchmarked against MiniMax-Speech and ElevenLabs Multilingual v2. Cross-lingual assessments also revealed significant error rate reductions, such as a 66% relative drop for Chinese-to-Korean synthesis compared to previous systems. Furthermore, the VoiceDesign model established new state-of-the-art scores among open-source models on InstructTTSEval for Description-Speech Consistency and Response Precision in both Chinese and English, even rivaling commercial solutions like Hume and Gemini on several metrics.
Key Innovations of Qwen3-TTS
- Complete Open-Source Suite: Qwen3-TTS is released under an Apache 2.0 license, providing a full stack for high-quality TTS, rapid 3-second voice cloning, and instruction-based voice design across 10 languages, powered by the 12Hz tokenizer family.
- Efficient Streaming Architecture: The Qwen3-TTS-Tokenizer-12Hz uses 16 codebooks at 12.5 frames per second, demonstrating robust PESQ, STOI, and UTMOS scores. It supports packetized streaming, delivering approximately 320 milliseconds of audio per packet, with impressive sub-120 millisecond first-packet latency for the 0.6B and 1.7B models.
- Specialized Model Variants: The release includes Base models for cloning and general TTS, CustomVoice models with nine curated speakers and style prompts, and a VoiceDesign model capable of generating novel voices directly from natural language descriptions, which can then be repurposed by the Base models.
- Superior Multilingual Quality: A multi-stage alignment pipeline, incorporating DPO, GSPO, and speaker fine-tuning, equips Qwen3-TTS with exceptionally low word error rates and high speaker similarity. It achieved the lowest WER in six of ten languages and the best speaker similarity across all ten compared evaluated systems, along with state-of-the-art zero-shot English cloning on Seed TTS.
This article is a rewritten summary based on publicly available reporting. For the original story, visit the source.
Source: MarkTechPost
