From Text to Vision: What Changes When an LLM Learns to See

TL;DR

A vision-language model (VLM) is a text-only LLM with just three additions: a frozen vision encoder that converts images to vectors, a tiny trainable projector that maps those vectors into the LLM’s embedding space, and a token-replacement step that swaps image placeholder embeddings with the projected vision features. After that swap, the transformer doesn’t know the difference — image tokens are processed identically to text tokens. The LLM itself is barely modified.

Prerequisite: This article builds on Visualizing Every Step of LLM Inference. The transformer, attention, MLP, and sampling are all the same — we only cover what vision adds.

What Changes — and What Doesn’t

Here’s the text-only LLM next to the vision-language model. The red-outlined components are the only additions:

Architecture Comparison

Left: The text-only LLM — tokenize, embed, transform, predict. 64M parameters.

Right: The VLM — the same LLM, plus three new components (highlighted). The transformer layers, LM head, and sampling are completely unchanged. Total: 160M parameters, but the 94.6M vision encoder is frozen and the projector is only 1.18M — so only ~65M parameters are trainable.

Component	Text LLM	VLM	Changed?
Tokenizer	✓	✓ (uses `<\|image_pad\|>` token)	Minimal
Embedding table	`[6400, 768]`	`[6400, 768]` (same)	No
Vision Encoder	—	SigLIP2 (94.6M, frozen)	New
Vision Projector	—	MLP (1.18M, trainable)	New
Token Replacement	—	Swap embeddings	New
Transformer Layers	8 layers	8 layers (identical)	No
LM Head	`[6400, 768]`	`[6400, 768]`	No
Sampling	temperature, top-p	Same	No

The insight: making an LLM “see” doesn’t require changing the LLM at all — you just need to translate images into a format the LLM already understands (768-dimensional vectors).

Component ①: Vision Encoder — Image to Patch Vectors

The vision encoder is a pre-trained SigLIP2 model (a ViT-B/32 architecture) that converts an image into a sequence of feature vectors. It stays completely frozen — no training needed.

Vision Encoder

The process:

Resize the input image to 256×256 pixels
Split into an 8×8 grid of patches, each 32×32 pixels → 64 patches
Encode each patch through the Vision Transformer → 64 vectors of 768 dimensions

Each of the 64 output vectors captures the visual information in its patch — color, texture, objects, edges. The top-left patch (sky) will have a very different vector than the center patch (yellow car).

The vision encoder has 94.6M parameters, but they’re all frozen — it was pre-trained on image-text matching and serves purely as a feature extractor. No gradient flows through it during VLM training.

Property	Value
Architecture	SigLIP2 (ViT-B/32)
Input	`[1, 3, 256, 256]` — RGB image
Patch size	32×32 pixels
Output	`[1, 64, 768]` — 64 patch vectors
Parameters	94.6M (frozen)

Component ②: Vision Projector — Translating to LLM Language

The vision encoder outputs 768-dimensional vectors, but they live in “SigLIP space” — a different representation than the LLM’s embedding space. The vision projector bridges this gap with a small MLP:

LayerNorm(768) → Linear(768→768) → GELU → Linear(768→768)

Vision Projector

Left: Raw vision features from SigLIP2 — these are in the vision encoder’s representation space.

Right: After the projector — mapped to the LLM’s embedding space.

Cosine similarity panel: How similar are the projected vision vectors to various text token embeddings? Individual patches show different similarity patterns — patches containing the car show higher similarity to “car” and “yellow” than patches showing the sky.

The projector is the key trainable bridge — just 1.18M parameters (LayerNorm + two Linear layers). Its job during training is to learn the mapping from “how SigLIP2 describes an image” to “how the LLM represents meaning.”

Think of it as a translator between two languages — the vision encoder speaks “SigLIP” and the LLM speaks “MiniMind.” The projector learns to translate between them.

Property	Value
Architecture	LayerNorm → Linear → GELU → Linear
Input/Output	`[64, 768]` → `[64, 768]`
Parameters	1.18M (all trainable)

Component ③: Token Replacement — The Embedding Swap

This is the most elegant part. The input text contains 64 <|image_pad|> placeholder tokens where the image goes:

<|im_start|>user↵ <|image_pad|>×64 ↵What is in this image?<|im_end|>...

After the text embedding lookup, those 64 positions all have the same generic embedding (they’re all token ID 12). The token replacement step swaps those generic embeddings with the projected vision vectors:

Token Replacement

Top (before): The 64 image_pad positions (marked between the dashed lines) all have the same generic embedding — a uniform stripe.

Bottom (after): Each of the 64 positions now contains a unique, rich vision feature vector from the projector. The stripe region is now full of varied, meaningful patterns.

After this swap, the transformer processes all 88 tokens (64 image + 24 text) identically. It doesn’t know which tokens came from an image and which came from text — they’re all 768-dimensional vectors in the same space.

Input Sequence

This is the core design principle of modern VLMs (LLaVA, Qwen-VL, InternVL all use this approach): images are just another “language” that gets translated into the LLM’s token embedding space.

How Attention Sees Image Tokens

Once image tokens are in the sequence, the transformer’s self-attention mechanism processes them like any other tokens. But how much attention do text tokens pay to image tokens?

Attention to Image

Left (Layer 0): In the early layer, image tokens (pink bars) attend fairly uniformly to each other and to text tokens. Text tokens (blue) also attend to image tokens — the model is gathering visual context.

Right (Layer 7): In the final layer, the pattern is more focused. The text tokens after the image region (positions 69+: “What is in this image?”) show strong attention to the image tokens — they’re “looking at” the image to answer the question.

Spatial attention map (far right): Which patches does the last text token attend to most? The attention is overlaid on the original image. Brighter regions receive more attention — the model focuses on the most informative parts of the image when generating its response.

Training: Two Stages to Teach Vision

Unlike the text-only LLM which trains everything together, the VLM uses a two-stage approach that carefully controls what gets updated:

Training

Stage 1: Pretrain (Image-Text Alignment)

Data: Image-caption pairs (e.g., “A yellow sports car on a road”) Trained: Only the vision projector (1.18M params) Frozen: Vision encoder + entire LLM

The goal is narrow: teach the projector to translate SigLIP2 features into vectors the LLM can understand. The LLM doesn’t need to change — it already knows language. The projector just needs to learn the mapping.

Stage 2: SFT (Visual Instruction Tuning)

Data: Visual Q&A conversations (e.g., “What color is this car?” → “The car is yellow…”) Trained: Projector + LLM first and last layers Frozen: Vision encoder + LLM middle layers (1-6)

Now the projector is aligned, and we fine-tune the LLM’s edge layers to better integrate visual information. The middle layers stay frozen to preserve the model’s language abilities — this prevents catastrophic forgetting where the model loses text skills while learning vision.

This selective freezing is the key trick: it takes only ~65M trainable parameters (out of 160M total) to add vision capabilities to a pre-existing LLM.

How the Model Learns Image-Word Relationships

The training data and label masking are the key to understanding how vision learning actually works. Let’s compare text-only training with vision training using concrete samples.

Text-only pretrain (from the training article) — the model learns from raw text, with ALL tokens as targets:

Data:  {"text": "AI is a field of computer science"}

Input:   [BOS]  AI    is     a     field   of    computer  science
Labels:   AI    is     a     field   of    computer  science  [EOS]
          ✅     ✅     ✅     ✅      ✅     ✅        ✅       ✅     ← ALL trained

Every token predicts the next — the model learns word-to-word relationships.

Vision pretrain — an image-caption pair. The image tokens and prompt are masked; ONLY the caption is trained:

Data:  {image_bytes: <yellow_car.jpg>, 
        conversations: [{user: "<image>\nDescribe this image."},
                        {assistant: "A yellow sports car on a coastal road."}]}

Input:   [special tokens] [img0 img1 ... img63] Describe this image. [special] A   yellow  sports  car ...
Labels:   -100             -100 × 64             -100     -100  -100   -100     A   yellow  sports  car ...
          (masked)         (masked)              (masked)                       ✅    ✅      ✅     ✅  ← ONLY caption trained

The model must predict “yellow”, “sports”, “car” — but the only source of that information is the 64 image tokens. The text prompt just says “Describe this image” with no hint about content. To predict “yellow” correctly, the transformer must attend to the image patches that contain the yellow car, and the gradient flows back through attention into the projector, teaching it: “when SigLIP sees these features, output vectors that make the LLM say ‘yellow car.’”

Vision SFT — visual Q&A conversations. Same masking rule (only assistant response trained), but now with questions and richer answers:

Data:  {image_bytes: <golden_armor.jpg>,
        conversations: [{user: "<image> How does the color contrast contribute to the image?"},
                        {assistant: "The gold armor against the white background creates
                                      a dramatic, majestic effect..."}]}

Input:   [special] [img×64] How does the color ... [special] The gold armor against ...
Labels:    -100     -100×64   -100  -100  -100       -100     The gold armor against ...
           (masked)           (masked)                        ✅  ✅   ✅     ✅      ← ONLY answer trained

The SFT data also includes ~230K pure-text conversations (no image, just an 8×8 black placeholder) to prevent the model from forgetting how to handle text-only queries.

	Text Pretrain	Vision Pretrain	Vision SFT
Samples	1.27M text lines	1.27M image-caption pairs	2.9M mixed
Images	None	~640K unique	~650K unique
Format	JSONL	Parquet (image + text)	Parquet (image + text)
Trained tokens	All	Caption only	Answer only
What learns	Entire LLM (64M)	Projector only (1.18M)	Projector + first/last LLM layers
Loss	Cross-entropy	Cross-entropy (same!)	Cross-entropy (same!)
Goal	Learn language	Image → word mapping	Visual reasoning & conversation

The fundamental insight: vision training uses the exact same loss function as text training (cross-entropy on next-token prediction). The only difference is that some of the input tokens carry image information instead of text. The projector learns by receiving gradients from caption/answer words that flow backwards through attention to the image tokens.

Training Data: By the Numbers

How much data does it take to teach a model to see?

	Vision Pretrain	Vision SFT
File	`pretrain_i2t.parquet`	`sft_i2t.parquet`
Total rows	1,274,698	2,904,511
Unique images	~640K	~650K
Captions per image	~2 (93% have EN + CN pair)	~4 (multiple Q&A per image)
Image-caption rows	1,274,698 (all)	592,912
Image Q&A rows	—	2,081,333
Pure text rows	0	230,266 (8%)
Language	52% EN, 48% mixed CN+EN	51% EN, 46% mixed, 3% CN-only
Image size	All 256×256 JPEG	All 256×256 JPEG
Source	ALLaVA-4V	ALLaVA-4V + text conversations

The pretrain data is elegantly simple: ~640K unique images, each described twice (once in English, once in Chinese). This gives the projector 1.27M gradient signals to learn the image-to-language mapping. Seeing the same image with captions in both languages also helps the projector learn language-agnostic visual representations.

The SFT data is larger and more varied: 2.9M rows covering image captions (~593K), visual Q&A (~2.08M), and pure-text conversations (~230K). The pure-text rows use a tiny 8×8 all-black placeholder image to keep the parquet format uniform — their purpose is to prevent catastrophic forgetting, ensuring the model retains its text-only conversation skills while learning vision.

Compared to the text-only LLM’s training data:

	Text-Only LLM	VLM
Pretrain samples	1,270,238 text lines	1,274,698 image-caption pairs
SFT samples	905,718 conversations	2,904,511 mixed (image + text)
Total images	0	~650K unique
Total training rows	2.18M	4.18M

The VLM sees roughly 2× the data of the text-only model, but the extra volume is primarily image-text pairs and visual Q&A — the core text data is comparable in size.

Parameters, Memory, and Compute

Params Compute

Parameters (left): The VLM is 2.5× larger than the text LLM, but most of the new parameters (94.6M) are the frozen vision encoder.

Memory (right): 319 MB vs 128 MB in float16. The vision encoder weights must be in memory even though they never update during training.

Compute (right): The vision encoder and projector add compute only once per image (during the first forward pass). After that, the 64 vision tokens are processed identically to text tokens through the transformer. For multi-turn conversations about the same image, the vision overhead is amortized.

	Text LLM	VLM	Overhead
Total params	63.9M	159.7M	+95.8M (vision)
Trainable params	63.9M	65.1M	+1.2M (projector)
Memory (float16)	128 MB	319 MB	+191 MB
Tokens per image	0	64	+64 per image
Extra FLOPs	—	Vision encoder + projector (once per image)	Small
Inference speed	~51 tok/s	~50 tok/s	Negligible

Summary

Making an LLM see requires surprisingly little change:

Text LLM:                          VLM:

"What is AI?"                      🖼️ Image + "What is this?"
      │                                  │
      │                            ┌─────▼──────────┐
      │                            │ Vision Encoder  │ ← frozen SigLIP2 (94.6M)
      │                            │ Image → 64 vecs │   just a feature extractor
      │                            └─────┬──────────┘
      │                            ┌─────▼──────────┐
      │                            │ Vision Projector│ ← tiny MLP (1.18M)
      │                            │ 768 → 768      │   "translates" to LLM space
      │                            └─────┬──────────┘
      │                                  │
      ▼                                  ▼
  Embedding                         Embedding + REPLACE 64 positions
      │                                  │
      ▼                                  ▼
  8× Transformer ─────────────── 8× Transformer (IDENTICAL!)
      │                                  │
      ▼                                  ▼
  LM Head → token ──────────────── LM Head → token (IDENTICAL!)

The transformer doesn’t know the difference between image tokens and text tokens. That’s the entire trick — convert images into the same format the LLM already processes, and let the existing architecture handle the rest. No attention modifications, no special cross-modal layers, no architectural changes to the core model.

Self-Check

Why does the vision encoder stay frozen? What would happen if you fine-tuned it?
The projector maps 768 → 768 (same dimensionality). Why not skip it and feed SigLIP2 features directly?
Why freeze the middle LLM layers during SFT but unfreeze the first and last?
If you wanted to support higher-resolution images (512×512), how would the token count change? What about compute?
This model uses 64 image tokens. GPT-4V uses ~1000+ image tokens. What are the tradeoffs of more vs fewer image tokens?

References

Liu et al., “Visual Instruction Tuning” (LLaVA, 2023) — The architecture pattern used here
Zhai et al., “SigLIP: Sigmoid Loss for Language Image Pre-Training” (2023) — The vision encoder
Dosovitskiy et al., “An Image is Worth 16×16 Words” (ViT, 2020) — Vision Transformer foundation
MiniMind-V GitHub — The model used in this article
Visualizing Every Step of LLM Inference — Companion article on the text-only LLM
MoE Inference Visualized — How expert routing works