Mixed Precision Deep Dive

Why mixed precision is more than changing a dtype

Mixed precision is often summarized as “store the model in fewer bits and use FP32 where accuracy matters.” That description is directionally correct, but it leaves out the decisions that make a real implementation work:

1. Which format stores the weights, activations, gradients, and optimizer state?
2. Which hardware unit executes each operation?
3. What precision is used for multiplication and accumulation?
4. How are FP8 or FP4 values scaled into their very small representable ranges?
5. Which layers are safe to quantize, and which should remain in BF16 or FP32?
6. What must be stored in a checkpoint so that the model can be reconstructed?

The key idea is that a tensor’s storage dtype is only one part of its numerical contract. A modern low-precision linear layer may read packed FP4 weights, quantize a BF16 activation to FP4, execute the matrix multiplication on Tensor Cores, accumulate into FP32, and return BF16 to the residual stream.

This post builds that pipeline from the format upward.

---

1. Start with the numerical formats

A binary floating-point value has three logical fields:

sign | exponent | fraction

The exponent controls dynamic range. The fraction, together with the implicit leading bit for normal values, controls precision. More exponent bits let a format represent very large and very small values; more fraction bits let it distinguish nearby values.

Format	Sign	Exponent	Fraction	Effective precision	Approximate maximum finite value
FP32	1	8	23	24 bits	$3.4 \times 10^{38}$
TF32	1	8	10	11 bits	FP32 range
BF16	1	8	7	8 bits	FP32-like range
FP16	1	5	10	11 bits	65,504
FP8 E4M3	1	4	3	4 bits	448
FP8 E5M2	1	5	2	3 bits	57,344
FP4 E2M1	1	2	1	2 bits	6 before scaling

The exact handling of infinity, NaN, subnormal values, and saturation depends on the format variant and hardware API. The table is most useful as a model of the range-versus-precision tradeoff.

FP32

FP32 remains the reference format for many reductions, accumulators, optimizer states, statistics, and numerically sensitive operations. It is not always the storage or multiplication format because moving and multiplying 32-bit values costs more memory bandwidth, capacity, and energy.

FP16

FP16 has 11 bits of effective significand precision, but only 5 exponent bits. Its limited range makes overflow and underflow a practical concern during training. Traditional automatic mixed precision therefore combines FP16 compute with FP32 accumulation, FP32 master weights, and loss scaling.

BF16

BF16 gives up fraction bits to retain the 8-bit exponent of FP32. It is less precise than FP16 for values of similar magnitude, but its much larger range usually makes training easier. This is why BF16 is a common default storage and non-GEMM compute format for modern large-model training.

TF32

TF32 uses FP32’s exponent range with an FP16-sized significand. It is primarily a Tensor Core arithmetic mode for FP32 matrix operations, not a compact tensor storage format: inputs and outputs are still normally stored as FP32.

TF32 accelerates models that expect FP32 range and interfaces without obtaining full FP32 multiplication precision. Accumulation is typically FP32.

FP8 E4M3 and E5M2

FP8 needs scaling because eight bits cannot provide both wide range and fine precision.

• E4M3 spends more bits on the fraction. It is usually preferred for forward-pass weights and activations.
• E5M2 spends more bits on the exponent. Its additional range can be useful for gradients, which are more likely to contain outliers.

NVIDIA Transformer Engine’s classic hybrid FP8 recipe reflects this tradeoff: E4M3 for the forward pass and E5M2 for the backward pass. Newer recipes can use more granular block scaling rather than one scale for an entire tensor.

NVFP4

The underlying NVFP4 element is E2M1: one sign bit, two exponent bits, and one fraction bit. Its unscaled values are drawn from a tiny set around:

0, 0.5, 1, 1.5, 2, 3, 4, 6

and their negatives. That value set is far too limited for a whole weight matrix by itself. NVFP4 becomes practical through a two-level scaling scheme:

1. every block of 16 values shares an FP8 E4M3 scale;
2. the tensor also has an FP32 scale that brings its overall range into the range covered by those block scales.

Conceptually, reconstruction is:

\[\hat{x_i} = q_i \times s_{\text{block}} \times s_{\text{tensor}},\]

where $q_i$ is the 4-bit E2M1 value.

The E4M3 block scale is not restricted to a power of two, so it can fit a local distribution more closely than a power-of-two scale. The 16-value block is also small enough to prevent one outlier from setting the scale for a large region of the tensor.

NVFP4 therefore costs slightly more than four bits per value. Ignoring padding and layout metadata, 16 values require 64 data bits plus one 8-bit block scale, or 4.5 bits per value. The additional FP32 tensor scale is negligible for a large tensor.

---

2. Storage, multiplication, accumulation, and output are separate choices

Calling an operation an “FP8 GEMM” is incomplete. A useful precision contract looks like:

weight storage:        FP8 E4M3
activation storage:    BF16
activation at GEMM:    FP8 E4M3
multiplication:        FP8 Tensor Core path
accumulation:          FP32
epilogue output:       BF16

The accumulator is especially important. For

\[C_{m,n} = \sum_{k=0}^{K-1} A_{m,k}B_{k,n},\]

every output contains $K$ products and $K-1$ additions. Accumulating thousands of terms directly in FP8 or FP4 would destroy too much information. Low-bit inputs are useful because they reduce storage, bandwidth, and multiplication cost; the sum normally needs a wider format.

The output dtype is another independent decision. An FP32 accumulator may be converted to BF16 before it is written to memory. That final conversion loses precision even though the internal accumulation was FP32.

This is the general pattern behind mixed precision:

compact storage
  -> scaled low-precision operands
  -> high-throughput multiplication
  -> wider accumulation
  -> high-precision epilogue where needed
  -> compact output

---

3. CUDA Cores and Tensor Cores have different jobs

NVIDIA GPUs contain general-purpose CUDA Cores and specialized Tensor Cores. Mixed-precision software normally uses both in one fused kernel.

Hardware	Typical work
CUDA Cores	Addressing, elementwise arithmetic, type conversion, scaling, normalization, activation functions, reductions, and general FP32/FP16/BF16 code
Tensor Cores	Dense matrix multiply-accumulate operations using supported low-precision input formats and wider accumulators

CUDA Cores can execute ordinary floating-point instructions, and some architectures have packed lower-precision instructions. Tensor Cores are different: they operate on matrix fragments and deliver much higher throughput for GEMM-shaped work.

A fused low-precision linear kernel can therefore:

1. load BF16 activations;
2. compute an activation maximum and scale using CUDA Core or specialized scaling instructions;
3. convert and pack the activations to FP8 or FP4;
4. execute the matrix multiply on Tensor Cores;
5. accumulate in FP32;
6. apply bias and an activation function in the epilogue;
7. write BF16 output for the next residual operation.

Precision support evolves by GPU generation

The important progression is:

• Volta introduced Tensor Cores for FP16 matrix operations.
• Ampere expanded Tensor Core use with BF16 and TF32.
• Hopper added native FP8 Tensor Core support.
• Blackwell adds fifth-generation Tensor Cores with FP4, including NVFP4, as well as additional low-precision choices.

Not every NVIDIA GPU supports every format. Casting a tensor to a smaller dtype does not guarantee that an application uses the desired Tensor Core path. The library, kernel, shapes, strides, alignment, and GPU generation must all support that path.

Tensor Cores also do not replace CUDA Cores. Softmax, normalization, scaling, indexing, and many reductions are not matrix multiplications. A well-optimized model keeps those operations in an appropriate higher precision and uses Tensor Cores for the GEMMs that dominate runtime.

---

4. Why FP8 and NVFP4 require explicit scaling

For a low-precision format with largest finite magnitude $Q_{\max}$, a simple symmetric quantizer chooses:

\[s = \frac{\max_i |x_i|}{Q_{\max}},\]

then computes:

\[q_i = \operatorname{clip}\left( \operatorname{round}\left(\frac{x_i}{s}\right), -Q_{\max}, Q_{\max} \right).\]

The approximate reconstructed value is:

\[\hat{x_i} = s q_i.\]

The scale decides which part of the original distribution the low-precision format covers. A scale that is too small causes saturation. A scale that is too large avoids saturation but maps many small values to zero.

Unlike common INT8 quantization, FP8 and FP4 GEMM formats are usually used with symmetric scaling and do not need an integer zero point. Their encoded values already have a sign and exponent.

Per-tensor scaling

One scale covers the complete tensor. This is simple and has little metadata overhead, but a single outlier can reduce precision for millions of ordinary values.

Per-channel scaling

Each output channel or input channel gets its own scale. This is common for weight-only quantization because weights are static and channel scales can be computed offline.

Per-token scaling

Each activation token gets its own scale. This follows variation between tokens better than one tensor-wide scale and is useful when activation magnitudes change with the input.

Block scaling

Small blocks share a scale. Block scaling is more expensive in metadata and scale management, but it handles local variation much better. NVFP4 uses one E4M3 scale for every 16 elements, plus its tensor-level FP32 scale.

The scale granularity is as important as the element format. A well-scaled FP4 tensor can preserve a model better than a badly scaled FP8 tensor.

---

5. FP8 runtime scaling: delayed, current, and block-scaled

Activations change for every batch, so their scales cannot always be frozen in the checkpoint.

Delayed scaling

The kernel records the activation’s absolute maximum, often into an amax history. A later iteration derives a scale from that history.

step N:     quantize with an existing scale and record amax
step N+1:   update the scale using recent amax values

Delayed scaling avoids putting a global reduction directly on the critical path of the current GEMM. The tradeoff is that the chosen scale describes recent tensors rather than the exact current tensor. Sudden distribution shifts can therefore cause clipping or wasted range.

For exact training resumption, the current scales and any amax history are part of the training state, just like the optimizer and random-number-generator state.

Current scaling

Current scaling computes the scale from the tensor being consumed now. It tracks changing activation distributions better, but the amax calculation must be efficient. Modern kernels fuse or pipeline the reduction, conversion, and GEMM to avoid writing an extra quantized activation tensor to global memory.

Block scaling

Block-scaled formats replace one global decision with many local decisions. This greatly reduces outlier sensitivity. Hardware and libraries must agree on the block orientation, block size, packed layout, and scale representation, so block-scaled checkpoints are not just arrays of 4-bit or 8-bit values.

---

6. The practical checkpoint strategy

The most robust workflow keeps two conceptually different artifacts.

6.1 Canonical training or source checkpoint

Keep a portable BF16, FP16, or FP32 checkpoint as the source of truth. For continued training, it normally includes:

• BF16/FP16 model weights or FP32 master weights;
• optimizer states, often in FP32 or an optimizer-specific compressed format;
• gradient scaler state when FP16 loss scaling is used;
• scheduler, training step, RNG, and distributed-training state;
• FP8 scale and amax history if exact continuation of an FP8 recipe matters.

Do not save transient quantized activations. They depend on runtime inputs and should be regenerated.

Keeping a canonical checkpoint matters because quantization is deployment specific. A model calibrated for FP8 on Hopper may later be exported to NVFP4 for Blackwell, or recalibrated with a better recipe. If the only remaining artifact is aggressively quantized, that conversion starts from already lost information.

6.2 Deployment checkpoint

Create a separate checkpoint optimized for a specific runtime and hardware target. It should contain:

• packed quantized weights;
• every scale required to reconstruct those weights;
• format identifiers such as E4M3, E5M2, or NVFP4 E2M1;
• scale granularity and block shape;
• tensor shapes, padding, packing order, and kernel layout;
• the original or output dtype expected by the surrounding graph;
• a list of layers intentionally kept in BF16 or FP16;
• calibration and quantization recipe metadata.

For an FP8 weight tensor, the checkpoint usually stores one byte per weight plus per-tensor, per-channel, or block scales.

For an NVFP4 weight tensor, it stores two E2M1 values per byte, one E4M3 scale per 16-value block, and the FP32 tensor-level scale. The layout and scale metadata are required data, not optional annotations.

Conceptually:

model.layers.0.mlp.up_proj.weight.data
model.layers.0.mlp.up_proj.weight.block_scales
model.layers.0.mlp.up_proj.weight.tensor_scale
model.layers.0.mlp.up_proj.weight.quantization_format = "NVFP4"
model.layers.0.mlp.up_proj.weight.block_size = 16

In practice, use the exporter provided by the serving stack rather than hand-packing tensors. NVIDIA TensorRT Model Optimizer and LLM Compressor can produce quantized checkpoints for runtimes such as TensorRT-LLM and vLLM. Framework-defined checkpoint formats also preserve the metadata needed to select compatible kernels.

---

7. What happens during runtime

A practical inference path separates static weight preparation from dynamic activation scaling.

Step 1: load static quantized weights

Weights do not change during inference. Their FP8 or NVFP4 values and scales can be computed offline and loaded directly into the layout expected by the GEMM kernel.

Repacking at model load time is acceptable when converting from a portable checkpoint to a device-specific layout, but repeatedly quantizing weights on every request would waste time.

Step 2: keep the residual stream in BF16 or FP16

The input to a Transformer block is typically BF16 or FP16. This gives normalization, residual addition, and elementwise operations substantially more range and precision than FP8 or FP4.

Step 3: dynamically scale the activation at each quantized GEMM

Immediately before a quantized linear operation:

1. measure the relevant activation range;
2. derive per-tensor, per-token, or per-block scales;
3. convert and pack the activation;
4. feed the quantized activation, quantized weight, and both sets of scales to the Tensor Core kernel.

Good implementations fuse these steps or overlap them so that a full quantized activation is not unnecessarily written to and read from HBM.

Step 4: multiply low, accumulate high

Tensor Cores consume the scaled low-precision operands. Partial products are accumulated in a wider type, commonly FP32, because the reduction dimension can contain thousands of terms.

Step 5: return to the residual precision

The epilogue can apply bias, gating, or an activation function before converting the result back to BF16 or FP16. Residual connections then combine values in that wider format.

The complete path is:

BF16 activation
  -> dynamic scale and quantize
  -> FP8 or NVFP4 Tensor Core GEMM with quantized weights
  -> FP32 accumulation
  -> fused epilogue
  -> BF16 output
  -> BF16 residual add

Low precision is therefore usually an execution format around GEMMs, not the dtype of every intermediate tensor in the model.

---

8. Which Transformer layers are normally quantized?

The highest return comes from quantizing the large linear layers because they contain most of the parameters and FLOPs.

Component	Typical precision choice	Why
Attention Q, K, V projections	FP8 or FP4 GEMM	Large, regular matrix multiplications
Attention output projection	FP8 or FP4 GEMM	Large, regular matrix multiplication
MLP gate/up/down projections	FP8 or FP4 GEMM	Usually the largest compute and parameter cost
MoE expert matrices	FP8 or FP4 GEMM	Very large aggregate weight footprint
Embedding table	BF16/FP16 initially; quantize after validation	Lookup-bound and sometimes sensitive; benefit depends on memory pressure
Final language-model head	BF16/FP16 initially; quantize after validation	Directly affects logits and may share weights with embeddings
RMSNorm/LayerNorm	BF16 with FP32 statistics	Reduction and normalization are sensitive to error
Softmax	BF16/FP32	Exponentials and probability normalization are sensitive
Residual additions	BF16/FP16	Small updates can disappear in very low precision
RoPE and positional math	BF16/FP32	Elementwise trigonometric and rotation operations
Loss and metric reductions	FP32	Need stable global reductions
Optimizer states and master weights	Usually FP32 or a validated optimizer format	Tiny update errors can accumulate across training

This is a conservative starting point, not a permanent rule. Well-calibrated deployments may quantize embeddings, the language-model head, and nearly every linear layer. Smaller or unusually sensitive models may need more BF16 fallbacks.

Two useful principles are:

1. quantize GEMMs first;
2. preserve high-precision islands around reductions, normalization, probabilities, residual paths, and training state.

What about the KV cache?

The KV cache is a separate quantization decision. It can dominate memory during long-context inference, so FP8 KV-cache storage can be valuable. Its scale granularity, accuracy impact, and supported attention kernel must be validated independently from weight quantization. A model with FP8 weights does not automatically have an FP8 KV cache.

---

9. A practical FP8 training recipe

A common mixed-precision training design is:

model parameters:          BF16
forward GEMM operands:     FP8 E4M3
backward GEMM operands:    FP8 E5M2 or recipe-specific FP8
GEMM accumulation:         FP32
residual stream:           BF16
normalization statistics:  FP32
optimizer/master state:    FP32 or validated compressed state

The BF16 parameters remain the trainable source of truth. They are quantized to FP8 for selected GEMMs and updated from gradients in a wider precision. The model is not repeatedly updating an FP8 weight as if it were a normal high-precision parameter.

The exact implementation may cache quantized weights between operations or iterations when it can prove they have not changed. Scale updates, weight updates, distributed synchronization, and recomputation determine whether that cache is worthwhile.

For training, start by converting the large linear layers and leave normalization, softmax, loss computation, optimizer state, and the residual stream in BF16/FP32. Then use loss curves, gradient statistics, downstream evaluation, and overflow/saturation telemetry to decide whether more operations can be lowered safely.

---

10. A practical FP8 or NVFP4 inference recipe

For inference, weights are static, so post-training quantization is often the first approach:

1. start from the canonical BF16/FP16 checkpoint;
2. run representative calibration data;
3. quantize the large attention, MLP, and expert matrices;
4. store packed weights and their scale metadata;
5. keep sensitive layers in BF16;
6. dynamically quantize activations at runtime;
7. evaluate model quality, not just layer-wise reconstruction error;
8. benchmark end-to-end latency, throughput, memory, and power.

If post-training quantization loses too much quality, use quantization-aware training or a recovery/fine-tuning method. The goal is not to force every tensor into four bits. The goal is to obtain the best end-to-end system under an explicit accuracy budget.

NVFP4 is most practical when all of the following are true:

• the target is Blackwell-class hardware with native support;
• the runtime has NVFP4 kernels for the model architecture;
• the checkpoint exporter and runtime agree on packing and scale layout;
• representative calibration or fine-tuning confirms acceptable quality;
• the workload is large enough for GEMM and memory savings to dominate scaling overhead.

Without native support, expanding FP4 values into a wider format before every operation can erase much of the intended performance benefit.

---

11. Common mistakes

Saving only the quantized deployment checkpoint

This removes the best source for future training, recalibration, or export to a different hardware format. Keep the canonical higher-precision checkpoint.

Saving low-bit values without scales

FP8 and NVFP4 tensors are not meaningful without their scale metadata. For NVFP4, both block scales and the tensor-level scale are required.

Quantizing the whole graph uniformly

Linear layers, normalization, softmax, residual addition, and optimizer updates have different numerical behavior. One dtype for the complete graph is usually neither the fastest nor the most accurate design.

Assuming a cast guarantees Tensor Core execution

Hardware generation, dimensions, alignment, layout, library configuration, and kernel availability determine whether Tensor Cores are actually used.

Measuring only checkpoint size

End-to-end speed also depends on activation conversion, scale reductions, collective communication, attention, memory layout, and CPU scheduling. A smaller checkpoint is useful, but it does not prove lower latency.

Using one scale for a distribution with severe outliers

The outlier consumes the representable range and pushes ordinary values toward zero. Use per-channel, per-token, or block scaling, or keep the sensitive tensor in a wider format.

---

12. The mental model

Mixed precision is easiest to reason about as a sequence of precision boundaries:

canonical checkpoint:
    BF16/FP16/FP32 weights

deployment export:
    quantize static weights
    + save packed values
    + save complete scale and layout metadata

runtime:
    keep residual activations in BF16/FP16
    -> dynamically scale near each GEMM
    -> multiply in FP8 or NVFP4 on Tensor Cores
    -> accumulate in FP32
    -> convert back to BF16/FP16

sensitive islands:
    normalization, softmax, residuals, reductions,
    loss, master weights, and optimizer state

The low-bit format receives most of the attention, but the complete system is defined by the scale granularity, accumulator precision, checkpoint metadata, hardware support, and placement of high-precision islands.

FP8 and NVFP4 work not because eight or four bits are universally sufficient, but because modern GPU software uses those bits exactly where they provide the largest matrix-compute and memory benefit, then returns to wider precision before numerical error becomes unacceptable.

---

References

• NVIDIA CUDA Programming Guide
• NVIDIA Tensor Cores
• NVIDIA Transformer Engine
• NVIDIA: Introducing NVFP4 for Efficient and Accurate Low-Precision Inference
• NVIDIA TensorRT Model Optimizer
• LLM Compressor