Navigation
⚡ Inference Systems· 8 min read

Speech Models

ASR chunked streaming (Whisper), TTS real-time factor, voice cloning pipelines — serving audio models at scale.

speechasrttswhispervoice-cloningstreamingaudio

Speech inference spans two directions: speech-to-text (ASR) and text-to-speech (TTS). Each has unique latency requirements and serving patterns that differ from both LLM and diffusion serving.

1. ASR: Speech-to-Text

Whisper Architecture

Whisper is an encoder-decoder model:

Audio → Mel Spectrogram → Encoder (compute-heavy, one pass)
  → Decoder (autoregressive, like LLM decode)
  → Text tokens → Detokenize → Transcript

Encoder (1 pass): Processes 30 seconds of audio in ~0.5s on H100. Compute-bound, benefits from batching.

Decoder (autoregressive): Generates text tokens one at a time. Memory-bound like LLM decode, but much shorter sequences (100-300 tokens for 30s audio).

Chunked Streaming

For real-time transcription, audio is split into overlapping chunks:

Audio stream: [────────────────────────────────────]
Chunk 1:      [═══════]                              → transcribe
Chunk 2:           [═══════]                         → transcribe
Chunk 3:                [═══════]                    → transcribe
               ↑ overlap ↑

Overlap ensures words at chunk boundaries aren’t missed. Typical: 30s chunks with 5s overlap.

Latency = chunk_duration + encode_time + decode_time. For streaming: user sees transcription ~1-2s after speaking.

Serving at Scale

OptimizationEffect
Batch multiple audio streamsEncode in parallel (high throughput)
FP16 encoder2× faster encode
Speculative decoding for decoderFaster text generation
Dedicated encoder + decoder poolsEncoder compute-heavy, decoder memory-heavy

2. TTS: Text-to-Speech

Architecture Variants

ArchitectureHow it worksLatencyQuality
Autoregressive (VALL-E style)Generate audio tokens like LLM generates textHigher (sequential)High (natural prosody)
Non-autoregressive (VITS, FastSpeech)Generate all frames in parallelLower (single pass)Good (less natural)
Diffusion-based (Voicebox)Iterative denoising on mel spectrogramsMedium (20-50 steps)Very high
Hybrid (F5-TTS, CosyVoice)Flow matching + vocoderMediumHigh

Real-Time Factor (RTF)

RTF=generation timeaudio duration\text{RTF} = \frac{\text{generation time}}{\text{audio duration}}

RTF < 1 is required for real-time serving. RTF = 0.3 means generating 10s of audio takes 3s.

ModelRTF (H100)Streaming?
VITS0.05✅ (non-AR, instant)
CosyVoice0.2-0.5✅ (chunk-by-chunk)
VALL-E0.5-1.0⚠️ (AR, depends on length)

Voice Cloning Pipeline

Reference audio (3-10s) → Speaker encoder → Speaker embedding
Text prompt → TTS model (conditioned on embedding) → Generated audio

Serving consideration: Speaker embeddings can be cached — clone once, generate many times. The embedding is tiny (~256 floats) compared to the model.

3. Common Patterns for Audio Serving

PatternDescription
Streaming I/OWebSocket for real-time audio streaming (both ASR and TTS)
Chunked processingProcess audio in fixed-size chunks for bounded latency
Vocoder sharingMultiple TTS requests share the vocoder (mel → waveform is generic)
GPU/CPU splitEncoder/model on GPU, pre/post-processing on CPU

Self-Check

  • Explain Whisper’s encoder-decoder architecture and why the encoder is compute-bound
  • Describe chunked streaming for real-time ASR
  • Define RTF and calculate it for a given model
  • Compare autoregressive vs non-autoregressive TTS for latency
  • Design a voice cloning serving pipeline with cached speaker embeddings

References