站内导读
定位:微软开源的语音 AI 项目,定位前沿语音 AI,同时覆盖语音合成与语音识别两个方向。 核心能力:包含文字转语音(TTS)与自动语音识别(ASR)模型,并配套了技术报告、在线试玩页面和 Colab 演示,方便直接上手体验和评估效果。 适用人群:做语音产品、播客或有声内容的开发者,以及研究语音生成与识别的工程团队。项目使用 Python 开发,GitHub 上已有 5.5 万余 star。
位置づけ:Microsoft がオープンソースで公開する音声 AI プロジェクトで、フロンティア音声 AI を掲げ、音声合成と音声認識の両方向をカバーします。 主な機能:テキスト読み上げ(TTS)と自動音声認識(ASR)のモデルを含み、技術レポート、オンラインの試用ページ、Colab デモが用意されているため、そのまま触って性能を評価できます。 対象ユーザー:音声プロダクト、ポッドキャスト、音声コンテンツを開発するエンジニアや、音声生成・認識を研究するチーム。Python 製で、GitHub では 5.5 万を超えるスターを集めています。
정의: Microsoft가 오픈소스로 공개한 음성 AI 프로젝트로, 프런티어 음성 AI를 표방하며 음성 합성과 음성 인식을 모두 다룹니다. 핵심 기능: 텍스트 음성 변환(TTS)과 자동 음성 인식(ASR) 모델을 포함하고, 기술 보고서와 온라인 체험 페이지, Colab 데모가 함께 제공되어 바로 사용해 보고 성능을 평가할 수 있습니다. 추천 대상: 음성 제품, 팟캐스트, 오디오 콘텐츠를 개발하는 엔지니어와 음성 생성·인식을 연구하는 팀. Python으로 개발되었으며 GitHub에서 5.5만 개가 넘는 스타를 받았습니다.
What it is: Microsoft's open-source voice AI project, positioned as frontier voice AI and covering both speech synthesis and speech recognition. Key features: It includes text-to-speech (TTS) and automatic speech recognition (ASR) models, along with technical reports, online playgrounds, and a Colab demo, so you can try it out and evaluate quality right away. Who it's for: Developers building voice products, podcasts, or audio content, and teams researching speech generation and recognition. Built with Python, it has over 55,000 stars on GitHub.
README · 详细文档
来源 · 查看原文
📰 News
2026-09-03: 🚀 We released VibeVoice-ASR-Streaming, a unified streaming ASR model that continuously transcribes ''who said what'' as speech arrives, with support for customized hotwords and 10 languages. [Demo] [Models] [Report]
2026-07-23: ⚡ We released VibeVoice-ASR-BitNet, an edge CPU inference engine for VibeVoice-ASR. Through heterogeneous quantization (I8S + I2S), the model is compressed from 4.62 GB to 1.58 GB with real-time inference (RTF < 1) on 3+ CPU threads — no GPU required. [Code] [Models] [Report]
2026-03-12: 🚀 VibeVoice-ASR is now integrated into Azure AI Foundry Labs! You can now explore and test our unified speech-to-text capabilities directly through Microsoft Foundry.
2026-03-06: 🚀 VibeVoice ASR is now part of a Transformers release! You can now use our speech recognition model directly through the Hugging Face Transformers library for seamless integration into your projects.
2026-01-21: 📣 We open-sourced VibeVoice-ASR, a unified speech-to-text model designed to handle 60-minute long-form audio in a single pass, generating structured transcriptions containing Who (Speaker), When (Timestamps), and What (Content), with support for User-Customized Context. Try it in Playground.
- ⭐️ VibeVoice-ASR is natively multilingual, supporting over 50 languages — check the supported languages for details.
- 🔥 The VibeVoice-ASR finetuning code is now available!
- ⚡️ vLLM inference is now supported for faster inference; see vllm-asr for more details.
- 🎤 Streaming recognition transcribes while the audio is still arriving, emitting text once per chunk; see asr-streaming, or vllm-asr-streaming to serve it.
- 📑 VibeVoice-ASR Technique Report is available.
2025-12-16: 📣 We added experimental speakers to VibeVoice‑Realtime‑0.5B for exploration, including multilingual voices in nine languages (DE, FR, IT, JP, KR, NL, PL, PT, ES) and 11 distinct English style voices. Try it. More speaker types will be added over time.
2025-12-03: 📣 We open-sourced VibeVoice‑Realtime‑0.5B, a real‑time text‑to‑speech model that supports streaming text input and robust long-form speech generation. Try it on Colab.
2025-09-05: VibeVoice is an open-source research framework intended to advance collaboration in the speech synthesis community. After release, we discovered instances where the tool was used in ways inconsistent with the stated intent. Since responsible use of AI is one of Microsoft’s guiding principles, we have removed the VibeVoice-TTS code from this repository.
2025-08-25: 📣 We open-sourced VibeVoice-TTS, a long-form multi-speaker text-to-speech model that can synthesize speech up to 90 minutes long with up to 4 distinct speakers. — accepted as an Oral at ICLR 2026! 🔥
Overview
VibeVoice is a family of open-source frontier voice AI models that includes both Text-to-Speech (TTS) and Automatic Speech Recognition (ASR) models.
A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details.
For more information, demos, and examples, please visit our Project Page.
| Model | Weight | Quick Try |
|---|---|---|
| VibeVoice-ASR-7B | HF Link | Playground |
| VibeVoice-ASR-Streaming | HF Link | Documentation |
| VibeVoice-ASR-BitNet (CPU) | HF Link | VibeASR.cpp |
| VibeVoice-TTS-1.5B | HF Link | Disabled |
| VibeVoice-Realtime-0.5B | HF Link | Colab |
Models
1. 📖 VibeVoice-ASR - Long-form Speech Recognition
VibeVoice-ASR is a unified speech-to-text model designed to handle 60-minute long-form audio in a single pass, generating structured transcriptions containing Who (Speaker), When (Timestamps), and What (Content), with support for Customized Hotwords.
- 🕒 60-minute Single-Pass Processing:
Unlike conventional ASR models that slice audio into short chunks (often losing global context), VibeVoice ASR accepts up to 60 minutes of continuous audio input within 64K token length. This ensures consistent speaker tracking and semantic coherence across the entire hour.
- 👤 Customized Hotwords:
Users can provide customized hotwords (e.g., specific names, technical terms, or background info) to guide the recognition process, significantly improving accuracy on domain-specific content.
- 📝 Rich Transcription (Who, When, What):
The model jointly performs ASR, diarization, and timestamping, producing a structured output that indicates who said what and when.
- 🎤 Streaming Recognition:
A streaming variant transcribes while the audio is still arriving, emitting text once per audio chunk rather than waiting for the recording to end.
📖 Documentation | 🤗 Hugging Face | 🎮 Playground | 🛠️ Finetuning | 📊 Paper | 🎤 Streaming


https://github.com/user-attachments/assets/acde5602-dc17-4314-9e3b-c630bc84aefa
2. 🎙️ VibeVoice-TTS - Long-form Multi-speaker TTS
Best for: Long-form conversational audio, podcasts, multi-speaker dialogues
- ⏱️ 90-minute Long-form Generation:
Synthesizes conversational/single-speaker speech up to 90 minutes in a single pass, maintaining speaker consistency and semantic coherence throughout.
- 👥 Multi-speaker Support:
Supports up to 4 distinct speakers in a single conversation, with natural turn-taking and speaker consistency across long dialogues.
- 🎭 Expressive Speech:
Generates expressive, natural-sounding speech that captures conversational dynamics and emotional nuances.
- 🌐 Multi-lingual Support:
Supports English, Chinese and other languages.
📖 Documentation | 🤗 Hugging Face | 📊 Paper
English
https://github.com/user-attachments/assets/0967027c-141e-4909-bec8-091558b1b784
Chinese
https://github.com/user-attachments/assets/322280b7-3093-4c67-86e3-10be4746c88f
Cross-Lingual
https://github.com/user-attachments/assets/838d8ad9-a201-4dde-bb45-8cd3f59ce722
Spontaneous Singing
https://github.com/user-attachments/assets/6f27a8a5-0c60-4f57-87f3-7dea2e11c730
Long Conversation with 4 people
https://github.com/user-attachments/assets/a357c4b6-9768-495c-a576-1618f6275727
3. ⚡ VibeVoice-Streaming - Real-time Streaming TTS
VibeVoice-Realtime is a lightweight real‑time text-to-speech model supporting streaming text input and robust long-form speech generation.
- Parameter size: 0.5B (deployment-friendly)
- Real-time TTS (~300 milliseconds first audible latency)
- Streaming text input
- Robust long-form speech generation (~10 minutes)
📖 Documentation | 🤗 Hugging Face | 🚀 Colab
https://github.com/user-attachments/assets/0901d274-f6ae-46ef-a0fd-3c4fba4f76dc
Contributing
Please see CONTRIBUTING.md for detailed contribution guidelines.
⚠️ Risks and Limitations
While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model (specifically, Qwen2.5 1.5b in this release). Potential for Deepfakes and Disinformation: High-quality synthetic speech can be misused to create convincing fake audio content for impersonation, fraud, or spreading disinformation. Users must ensure transcripts are reliable, check content accuracy, and avoid using generated content in misleading ways. Users are expected to use the generated content and to deploy the models in a lawful manner, in full compliance with all applicable laws and regulations in the relevant jurisdictions. It is best practice to disclose the use of AI when sharing AI-generated content.
We do not recommend using VibeVoice in commercial or real-world applications without further testing and development. This model is intended for research and development purposes only. Please use responsibly.