Emerging Formats
NVFP4, MXFP4 (Blackwell), microscaling — how sub-4-bit floating point works and what hardware support means for inference.
Beyond INT4 and FP8, the next frontier is 4-bit floating point with microscaling — block-level scale factors that let FP4 elements represent useful value ranges. Blackwell GPUs bring native hardware support.
1. FP8 Recap: Why Scaling Still Matters
FP8 has limited range ( for E4M3). External scaling maps the data into this range:
Even floating-point quantization needs calibrated scales — without them, data can underflow (map to zero) or overflow (saturate at ). The scale granularity (per-tensor, per-channel, per-block) determines accuracy.
2. FP4 and Microscaling
The E2M1 Data Format
FP4 (E2M1): 2 exponent bits, 1 mantissa bit, 1 sign bit. Only 8 representable values per sign:
E2M1 positive values: 0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0
(plus negatives and zero)With only 16 total values, raw FP4 is nearly useless for neural networks. Microscaling makes it work.
Microscaling: Block-Level Scale Factors
Group elements into small microblocks (16 or 32 elements), each with its own scale:
The block scale stretches or compresses the 16 FP4 values to cover the block’s actual range. Different blocks can represent completely different magnitudes.
3. NVFP4 vs MXFP4
| Property | MXFP4 (OCP standard) | NVFP4 (NVIDIA) |
|---|---|---|
| Data element | E2M1 (4-bit) | E2M1 (4-bit) |
| Block size | 32 elements | 16 elements |
| Block scale format | E8M0 (powers of 2 only) | E4M3 (finer-grained) |
| Additional scaling | None specified | + per-tensor FP32 scale |
| Effective bits | 4 + 8/32 = 4.25 bits/element | 4 + 8/16 = 4.5 bits/element |
NVFP4 Reconstruction
Three levels: tensor-level FP32 scale → block-level E4M3 scale → element-level E2M1 data. The two-level scaling gives NVFP4 better dynamic range than MXFP4’s single-level.
Trade-Off
NVFP4 uses more metadata (smaller blocks, richer scales) for better accuracy. MXFP4 is more compact. Actual quality and speed depend on the hardware implementation and kernel support.
4. Hardware Support
| GPU Generation | FP8 | FP4 |
|---|---|---|
| Ampere (A100) | No native | No |
| Ada (L40S, 4090) | FP8 via specialized kernels | No |
| Hopper (H100, H200) | Native FP8 Tensor Cores | No |
| Blackwell (B200) | Native FP8 | Native NVFP4/MXFP4 Tensor Cores |
Blackwell doubles effective throughput again: FP4 Tensor Cores compute at 2× the rate of FP8, which is already 2× the rate of BF16.
| Precision | B200 TFLOPS | Relative to BF16 |
|---|---|---|
| BF16 | ~2,250 | 1× |
| FP8 | ~4,500 | 2× |
| FP4 | ~9,000 | 4× |
5. When FP4 Makes Sense
| Scenario | FP4 viable? | Notes |
|---|---|---|
| Decode latency (small batch) | ✅ Less data to load | Similar benefit to INT4, but native hardware |
| Large batch throughput | ✅ 4× GEMM speedup | If quality holds |
| Quality-sensitive tasks | ⚠️ Must validate | 4-bit floating point loses information |
| Long-context KV cache | Possible with NVFP4 | Research frontier |
FP4 is still emerging. As of 2024, production deployments primarily use FP8 (Hopper) and INT4 (GPTQ/AWQ). FP4 will become practical as Blackwell hardware proliferates and quantization toolchains mature.
Self-Check
- Explain the E2M1 format: how many representable values, what range
- Explain microscaling: how block-level scales make FP4 usable
- Compare MXFP4 vs NVFP4: block size, scale format, effective bits
- Write the NVFP4 reconstruction formula (tensor scale × block scale × element)
- Explain Blackwell’s 4× throughput advantage for FP4 over BF16