The Omni Model: How a Tiny LLM Learns to Listen, See, Think, and Speak

TL;DR

An Omni model does everything: it hears (audio input), sees (image input), thinks (text reasoning), and speaks (streaming audio output) — all in a single unified model. MiniMind-O achieves this at just ~0.1B trainable parameters by adding a Talker module alongside the existing LLM (Thinker), connected via a hidden bridge. The Talker uses Multi-Token Prediction (MTP) to generate 8 parallel audio codebook streams that a frozen codec decodes into 24kHz streaming speech.

Prerequisite: This article builds on the companion articles covering the text LLM and the vision-language model. We focus on what Omni adds: audio input/output, the Thinker-Talker architecture, and speech generation.

The Evolution: Text → Vision → Omni

Each generation adds new modalities while keeping the previous architecture intact:

Evolution

MiniMind (text): Input text, output text. 64M parameters.

MiniMind-V (vision): Added a frozen vision encoder and projector. Input text + images, output text. 160M total.

MiniMind-O (omni): Added an audio encoder, a Talker module, and an audio codec. Input text + images + audio, output text + streaming speech. ~540M total, but only ~115M trainable.

The Thinker (the original LLM) barely changes. What’s new is everything around it.

Architecture: Thinker and Talker

The core innovation is splitting the model into two paths:

Architecture

Thinker (63.9M, 8 layers): The original MiniMind LLM. It processes all inputs — text tokens, injected audio features, injected vision features — and produces text output via the LM head. This is architecturally identical to the text-only model.

Talker (47.1M, 4 layers): A NEW, separate transformer module. It receives semantic conditioning from the Thinker’s middle layer (the “hidden bridge”) and generates 8 parallel streams of Mimi audio codebook codes. These codes are decoded by the frozen Mimi codec into a 24kHz audio waveform.

Hidden Bridge: The connection between Thinker and Talker. Instead of using the Thinker’s final layer (too shaped for next-token text prediction) or the embedding layer (too little semantic info), the bridge takes the output of Layer 3 (the middle) — which carries rich contextual and cross-modal information without being over-tuned by the LM head.

The Talker combines two signals:

h_{\text{talker}} = h_{\text{bridge}} \times \alpha_{\text{text}} + h_{\text{audio\_embed}} \times \alpha_{\text{audio}}

where $\alpha_{\text{text}}$ and $\alpha_{\text{audio}}$ are learned scalar scales, bridge is the Thinker’s semantic output, and audio_embed is the embedding of previously generated audio codes.

Why Two Separate Models?

A cascade system (ASR → LLM → TTS) adds latency, loses prosody, and can’t do real-time interruption. The Thinker-Talker design keeps everything end-to-end: the Talker generates speech tokens in lockstep with the Thinker’s text tokens, enabling streaming output without waiting for the full text response.

Multi-Token Prediction: 8 Codebook Layers at Once

Speech is represented using the Mimi codec — 8 codebook layers at 12.5 Hz (12.5 frames per second). Each frame has 8 integer codes (one per codebook), each in the range 0–2047.

The Talker uses MTP to predict all 8 codebook layers simultaneously, with a staggered delay:

MTP Codebooks

Top row: The Thinker’s text tokens — “Hi, Bro” followed by padding.

8 rows below: The Talker’s 8 codebook layers (L0–L7). Each cell contains an audio code (0–2047). Key details:

Staggered start: L0 begins generating at the first decode step. L1 starts one step later. L7 starts 7 steps later. This delay schedule means all 8 layers at a given column represent the same audio frame.
Stop tokens: Each layer independently generates a stop token (≥2048) when it’s done. Once all 8 layers have stopped, speech generation is complete.
Streaming: As soon as a complete column of 8 codes is available, the Mimi decoder can reconstruct that audio frame. Playback doesn’t wait for the full response.

The TalkerHead uses a shared base projection plus 8 lightweight per-codebook adapters:

TalkerHead: base_linear(768 → 2112) + 8 × adapter(768 → 256 → 2112)
Output for layer i = base(hidden) + adapter_i(hidden)

This keeps parameters low — a full independent head per codebook would cost 8× more.

Training Data: Teaching an Omni Model

The training data is the most complex in the series — each sample contains text, audio codes, speaker identity, and optionally raw audio:

Training Data

Stage 1: T2A (Text → Audio)

515K samples. Each contains:

conversations: Text Q&A (user question + assistant answer)
answer_audios: 8 lists of Mimi codes — the speech representation of the answer
ref_audios: Mimi codes of a reference voice (for voice style)
spk_emb: 192-dimensional speaker embedding vector

The model receives text input and must generate both text output (Thinker) and speech output (Talker). This stage teaches the Talker to produce coherent speech from the Thinker’s semantic conditioning.

Stage 2: A2A (Audio → Audio)

Adds question_audios — raw MP3 bytes of the user’s spoken question. Now the model receives spoken input (processed by the SenseVoice audio encoder + audio projector) and produces both text and speech output. This completes the hear → think → speak pipeline.

Comparison to Text and VLM Data

	Text SFT	VLM SFT	Omni T2A	Omni A2A
Text conversation	✅	✅	✅	✅
Image	—	✅ JPEG	—	—
Audio input	—	—	—	✅ MP3
Audio output (8 codebooks)	—	—	✅	✅
Speaker embedding	—	—	✅ (192-dim)	✅
Reference voice	—	—	✅ Mimi codes	✅ Mimi codes

Parameters: Where the Numbers Live

Parameters

Left: The trainable backbone (~115M). The Thinker is 63.9M (the original LLM), and the Talker adds 47.1M (4 transformer layers + MTP heads + audio embeddings). Projectors add ~2M.

Middle: The full model at inference (~540M). Three frozen external modules contribute 425M parameters: SenseVoice (234M), SigLIP2 (94.6M), and Mimi (96.2M). These are feature extractors/decoders that never train.

Right: Evolution across the three models. The trainable backbone grows from 64M → 65M → 115M (roughly 2× from text to Omni), but the frozen modules add significant overhead for inference memory.

Module	Params	Status
Thinker (8 layers)	63.9M	✅ Trainable
Talker (4 layers + MTP)	47.1M	✅ Trainable
Audio Projector	~1M	✅ Trainable
Vision Projector	~1M	✅ Trainable
Audio Encoder (SenseVoice)	234M	❄️ Frozen
Vision Encoder (SigLIP2)	94.6M	❄️ Frozen
Speech Codec (Mimi)	96.2M	❄️ Frozen
Total	~540M	~115M trainable

Inference: From Voice to Voice

Inference Flow

The complete inference pipeline:

User speaks → audio waveform captured
SenseVoice (frozen) → extracts speech features
Audio Projector → maps features to LLM embedding space
Thinker → processes text + audio + vision features through 8 transformer layers → generates text tokens
Hidden Bridge → Layer 3’s output feeds the Talker as semantic conditioning
Talker → combines bridge + previous audio codes → 4 transformer layers → 8 MTP heads → 8 streams of Mimi codes
Mimi Decoder (frozen) → converts 8 code streams to 24kHz audio waveform → streaming playback
Output → text and speech arrive simultaneously

Text and speech are generated in lockstep — the Thinker produces a text token while the Talker simultaneously produces audio codes for the same semantic content. This enables streaming: audio playback begins while the model is still generating.

Voice Cloning

The model supports voice cloning without fine-tuning. A reference audio clip is encoded into Mimi codes and a 192-dim speaker embedding (via CAM++). These are fed to the Talker as context — the speaker embedding is injected at a special <spk> token position, and the reference codes are prepended to the audio sequence. The Talker learns to match the voice style of the reference during generation.

Real-Time Barge-In

A Silero VAD (Voice Activity Detection) module runs alongside the model. When the user starts speaking while the model is generating, the VAD detects speech onset and triggers an interrupt — stopping generation and switching to listening mode. This enables near-duplex conversation.

Training Pipeline: Three Stages

Training Pipeline

Stage 1 (T2A): Full parameter training. The Thinker and Talker learn together — the Thinker processes text questions and produces semantic conditions, the Talker learns to convert those conditions into speech codes. LR = 5e-4.

Stage 2a (A2A — projector only): Only the audio projector trains. This aligns the SenseVoice speech features with the LLM’s embedding space, similar to the VLM’s vision projector alignment stage. LR = 5e-4.

Stage 2b (A2A — full): Full parameter training again at lower LR (2e-5). The complete hear → think → speak pipeline is fine-tuned end-to-end.

Throughout all stages, the three frozen modules (SenseVoice, SigLIP2, Mimi) never update.

What’s New vs the Text LLM

What's New

The left column shows everything shared with the text-only LLM — tokenizer, embedding table, transformer layers, attention, MLP, normalization, LM head, optimizer. These are unchanged.

The right column shows everything new in Omni — two frozen encoders, two projectors, the entire Talker module with MTP heads, the Mimi codec, speaker embedding support, and VAD for real-time interaction. The Omni model adds 11 new components to the text LLM’s foundation.

Summary

Text LLM:          input: text           → Thinker → output: text
VLM:               input: text + image   → Thinker → output: text  
Omni:              input: text + image + audio
                           ↓
                   Audio Encoder ─→ Audio Projector ─┐
                   Vision Encoder → Vision Projector ─┤
                   Text Tokenizer ────────────────────┘
                           ↓
                       THINKER (8 layers) → text output
                           ↓ (Layer 3 hidden bridge)
                       TALKER (4 layers, MTP) → 8× audio codes
                           ↓
                       Mimi Decoder → 24kHz streaming speech

The evolution from text to Omni follows a clear pattern: each stage adds new input/output modalities while keeping the core LLM (Thinker) largely intact. The Omni model’s key innovations are the Talker (a separate speech generation path), MTP (predicting 8 codebook layers simultaneously), and the hidden bridge (connecting Thinker semantics to Talker speech generation). Together, these enable a model that can hear, see, think, and speak — all at 0.1B trainable parameters.

Self-Check

Why does the Hidden Bridge use a middle layer (Layer 3) instead of the final layer? What information would be lost or distorted?
The Talker has 4 layers while the Thinker has 8. Why might fewer layers suffice for speech generation?
If the Mimi codec uses 12.5 Hz frame rate and the model generates at ~50 tokens/s, how does the staggered delay affect latency to first audio?
Why does Stage 2a train only the audio projector before Stage 2b trains everything? What would happen if you skipped 2a?
The TalkerHead uses a shared base + per-codebook adapters. What would be the parameter cost of 8 fully independent heads instead?

References

MiniMind-O Technical Report — Architecture details, training curves, and evaluation
Défossez et al., “Moshi: a speech-text foundation model for real-time dialogue” (2024) — Mimi codec and streaming speech
Ma et al., “Language Model Can Listen While Speaking” (2024) — Duplex voice interaction
SenseVoice — The audio encoder used
MiniMind GitHub — The base LLM
MiniMind-V GitHub — The vision-language extension
MiniMind-O GitHub — The Omni model