Rank #07 · Daily · AI
VibeVoice: Microsoft's Open-Source Frontier Voice AI Family
One Microsoft research umbrella for long-form speech recognition and lightweight realtime TTS — structured transcripts, streaming variants, and optional CPU inference.
What it does
VibeVoice is a family of open-source frontier voice AI models (TTS + ASR) from Microsoft. The README highlights a shared design idea: continuous acoustic/semantic speech tokenizers at an ultra-low 7.5 Hz frame rate, plus a next-token diffusion setup where an LLM handles text/dialogue context and a diffusion head fills in acoustic detail. Project page: microsoft.github.io/VibeVoice.
Active surfaces called out in the README (as of verification):
- VibeVoice-ASR — unified speech-to-text aimed at up to ~60 minutes of audio in a single pass, emitting structured Who (speaker) / When (timestamps) / What (content), with user-customized hotwords/context. Multilingual (README cites 50+ languages). Finetuning code, vLLM serving docs, and a playground are linked. Weights on Hugging Face; also noted as usable via Transformers and Azure AI Foundry Labs.
- VibeVoice-ASR-Streaming — transcribes while audio is still arriving (chunk emissions), with hotwords and multi-language support.
- VibeVoice-ASR-BitNet / VibeASR.cpp — edge CPU inference path with heterogeneous quantization; README claims compression from ~4.62 GB to ~1.58 GB and RTF < 1 on 3+ CPU threads (vendor figures — TidyHow has not reproduced).
- VibeVoice-Realtime-0.5B — lightweight realtime TTS with streaming text input; README cites ~300 ms first audible latency and ~10 minutes of robust long-form generation, plus experimental multilingual/style speakers. Colab demo available.
- VibeVoice-TTS-1.5B — historically the long-form multi-speaker TTS line (up to ~90 minutes / 4 speakers; ICLR 2026 Oral). Important: after misuse inconsistent with the research intent, Microsoft removed the VibeVoice-TTS code from this repository (announced 2025-09-05). Weights may still appear on Hugging Face tables, but the README marks TTS quick-try as disabled and frames the repo as research collaboration — do not treat TTS as a ready product drop from this GitHub tree.
Risks section is explicit: possible bias/inaccuracy from the base LLM, deepfake/disinformation risk from high-quality speech, and a recommendation against commercial/real-world use without further testing. Disclose AI-generated audio when sharing.
Who it's for
- Researchers and builders exploring long-form diarized ASR, streaming recognition, or CPU-side ASR compression.
- Teams evaluating open Microsoft voice models for demos, papers, or R&D — not drop-in production SaaS.
- Anyone who needs structured speaker-aware transcripts with hotwords, and can accept research-software caveats.
Less suitable if you need a turnkey commercial TTS product from this exact repo (TTS code was removed) or guaranteed production SLAs.
Stars / language / license
55,940 stars · Python · MIT
Data verified: 2026-11-06 09:09 Asia/Shanghai (GitHub API get_repository + README)
Trending note: +223 stars today (Trending selection captured 09:06 Asia/Shanghai)
Category tags
AI · voice · ASR · TTS · speech · Microsoft · HuggingFace · research
How it differs from similar projects
Compared with Whisper-class ASR. Whisper is strong general transcription; VibeVoice-ASR's pitch is hour-scale single-pass input plus joint speaker + timestamp + text structure and hotwords — a different product shape than short-chunk ASR.
Compared with hosted voice APIs (Azure Speech, ElevenLabs, etc.). Those are products with SLAs. VibeVoice is an open research family with explicit non-commercial-without-further-testing guidance.
Compared with other open TTS stacks (Piper, Coqui, StyleTTS, etc.). Realtime-0.5B is the lightweight streaming TTS path still documented here; the larger multi-speaker TTS line's source was pulled after misuse. Compare on licensing, model size, and whether code is actually in-tree.
Compared with diarization pipelines (pyannote + ASR glue). VibeVoice-ASR markets a unified Who/When/What model rather than a multi-stage stitch. Treat accuracy claims (DER / cpWER charts in the README) as author-reported — TidyHow has not re-run them.
Caveats. pushed_at on the main repo was 2026-10-08 at verification (activity quieter than daily-trending peers). Always re-check which sub-models still ship code vs weights-only.