Attention-FFN Separation

Attention and FFN have different compute characteristics — attention is memory-bandwidth-bound (reading long KV cache) while FFN is compute-bound (large matrix multiplications). Separating them onto specialized hardware is an emerging optimization.

1. The Asymmetry

Within each Transformer layer, attention and FFN have fundamentally different profiles:

Component	Arithmetic intensity	Bottleneck	Memory footprint
Attention	Low (reads KV cache: $O(s)$ per token)	Memory bandwidth	KV cache (grows with context)
FFN	High (large weight matrices)	Compute	Weights only (fixed)

For a 70B model at 8K context, batch=32:

Attention: Reads 32 × 8K × 320 bytes/token of KV = 80 MB per layer → bandwidth-limited
FFN: Computes 2 large GEMMs (h × 4h) → compute-limited

2. Why Separate?

On a single GPU, attention and FFN run sequentially — the GPU alternates between bandwidth-limited and compute-limited work. Neither fully utilizes its strength:

Single GPU timeline:
  [Attention: GPU compute 30% utilized, HBM 90%]
  [FFN: GPU compute 90% utilized, HBM 30%]
  [Attention: GPU compute 30% ...]
  → Average GPU utilization: ~60%

With separation:

Attention GPU (bandwidth-optimized):    [Attn] [Attn] [Attn] ...  90% HBM utilized
FFN GPU (compute-optimized):            [FFN]  [FFN]  [FFN]  ...  90% compute utilized
  → Each GPU runs the workload it's designed for

3. Implementation Approaches

Homogeneous Hardware, Different Configuration

Even with identical GPUs, you can optimize each pool differently:

Attention pool: More GPUs with smaller TP (maximize memory capacity for KV cache)
FFN pool: Fewer GPUs with larger TP (maximize compute throughput)

Heterogeneous Hardware

Use different hardware for each phase:

Component	Ideal hardware	Why
Attention	High-HBM GPUs (H200: 141 GB) or CPU+CXL	Need bandwidth + capacity for KV cache
FFN	High-compute GPUs (B200) or custom ASIC	Need TFLOPS for large GEMMs

Communication Between Attention and FFN

Each layer requires passing hidden states between attention and FFN:

T_{\text{transfer/layer}} = \frac{B \times h \times 2\text{bytes}}{\text{interconnect BW}}

For batch=32, $h=8192$ : $32 \times 8192 \times 2 = 0.5\text{MB}$ per layer. On NVLink: $0.5\text{MB} / 900\text{GB/s} \approx 0.6\mu\text{s}$ — negligible. On InfiniBand: still only $0.5\text{MB} / 50\text{GB/s} \approx 10\mu\text{s}$ .

The hidden state transfer is small compared to KV cache transfer in PD disaggregation — making attention-FFN separation more practical in terms of communication overhead.

4. Current Status

Attention-FFN separation is an emerging research direction, not yet production-standard:

Aspect	Status
Academic research	Active (several papers 2024-2025)
Production deployment	Limited (DeepSeek may use internally)
Engine support	Not in vLLM/SGLang mainline
Hardware support	CXL-attached memory enables new architectures

The concept is most relevant for:

Very long context (128K+) where KV cache dominates
Heterogeneous hardware deployments (mix of GPU types)
Custom silicon designs where attention and FFN accelerators differ

For PD disaggregation (the more established form of disaggregation), see Prefill-Decode Disaggregation.

Self-Check

Explain why attention is memory-bound and FFN is compute-bound
Calculate hidden state transfer overhead between attention and FFN GPUs
Compare attention-FFN separation with PD disaggregation
Identify workloads where attention-FFN separation provides the most benefit