Where gpu_memory_utilization Actually Goes: vLLM's Memory Budget, Line by Line

You set gpu_memory_utilization=0.9 and vLLM grabs 90% of the GPU. But 90% of what, measured when, and split how between weights, activations, and KV cache? I hit this concretely while benchmarking a Qwen3-235B reward server: at 0.92 on a 140 GiB H200, vLLM reported only 5.5 GiB of KV cache — the 110 GiB of MoE weights had eaten almost the entire budget. This post traces the one number from config field to the subtraction that decides your KV cache, so that result stops being surprising.

All vLLM permalinks are pinned to tag v0.18.0 (commit bcf2be96) of vllm-project/vllm so line numbers stay valid. Swap bcf2be96120005e9aea171927f85055a6a5c0cf6 for main in any link to see the latest. This is the V1 engine path.

The question, and the two misconceptions

gpu_memory_utilization is the knob everyone turns first and understands last. Two beliefs about it are both wrong:

1. “It’s a fraction of free memory.” No — it’s a fraction of total memory, evaluated against free only as a sanity check.
2. “It’s the KV cache size.” No — it’s the budget for weights + activations + non-torch + KV cache combined. KV cache is the remainder after the first three are measured and subtracted.

The whole mechanism is four files:

File	Role
vllm/config/cache.py	The config field — a per-instance fraction, default 0.9.
vllm/v1/worker/utils.py	request_memory() — turns the fraction into a byte budget: total × util.
vllm/utils/mem_utils.py	MemorySnapshot + memory_profiling — the three-category model of GPU memory.
vllm/v1/worker/gpu_worker.py	determine_available_memory() — the profiling run and the subtraction.

The one-line answer, which the rest of this post unpacks:

\[\text{KV cache} = \underbrace{\text{total} \times \texttt{gpu\_memory\_utilization}}_{\texttt{requested\_memory}} - \underbrace{(\text{weights} + \text{activation peak} + \text{non-torch})}_{\texttt{non\_kv\_cache\_memory}} - \text{cudagraph}\]

1. The config field: a per-instance fraction

CacheConfig.gpu_memory_utilization is one validated float:

# vllm/config/cache.py:41
gpu_memory_utilization: float = Field(default=0.9, gt=0, le=1)
"""The fraction of GPU memory to be used for the model executor, which can
range from 0 to 1. ... This is a per-instance limit, and only applies to the
current vLLM instance. It does not matter if you have another vLLM instance
running on the same GPU. For example, if you have two vLLM instances running
on the same GPU, you can set the GPU memory utilization to 0.5 for each
instance."""

The docstring quietly states the design decision that makes colocation possible: the limit is per-instance and vLLM makes no attempt to coordinate across instances. Two engines on one GPU at 0.5 each is your arithmetic to get right, not vLLM’s. (This is exactly the knob I had to split when colocating a 235B reward engine and a rollout engine on the same 8 GPUs — 0.5 for the reward, 0.3 for the rollout, and the remaining headroom for the FSDP actor. Get the sum wrong and one of them OOMs at startup.)

2. request_memory: total × util, validated against free

The fraction becomes bytes in request_memory(), called once during init_device before any weights load:

# vllm/v1/worker/utils.py:408-423
requested_memory = math.ceil(
    init_snapshot.total_memory * cache_config.gpu_memory_utilization
)

if init_snapshot.free_memory < requested_memory:
    raise ValueError(
        f"Free memory on device ... is less than desired GPU memory "
        f"utilization ({cache_config.gpu_memory_utilization}, "
        f"{format_gib(requested_memory)} GiB). Decrease GPU memory "
        f"utilization or reduce GPU memory used by other processes."
    )

return requested_memory

Two things to read out of this:

• It’s total_memory, not free. On a 140 GiB H200, 0.92 is ~129 GiB regardless of what else is on the card. If another process already holds 30 GiB, vLLM still wants 129 GiB of the total — it just measures whether it can have it.
• free is only a gate. If free < requested, you get the familiar “Free memory … is less than desired GPU memory utilization” error at startup. That’s a real ordering constraint in multi-engine setups: whoever initializes first eats free memory, and the second engine’s total × util may no longer fit under what’s left. The fix is lowering util, not increasing it.

So after this call vLLM holds a single number — requested_memory, a byte budget — and nothing has been allocated yet. Weights aren’t loaded; the KV cache doesn’t exist. The budget is a ceiling, and the rest of startup spends against it.

3. The three-category model: what “memory” even means here

Before the subtraction makes sense, you need vLLM’s mental model of a GPU’s memory, which lives in the memory_profiling docstring. It splits every byte on the device into three buckets:

# vllm/utils/mem_utils.py:197 (paraphrased from the docstring)
1. memory used by anything OTHER than this vLLM instance   (other processes)
2. memory used by torch IN this instance                   (weights + activations)
3. memory used in this instance but NOT by torch           (NCCL, attention-backend buffers, CUDA ctx)

MemorySnapshot.measure() is how vLLM reads those buckets off a live device:

# vllm/utils/mem_utils.py (measure(), condensed)
self.torch_peak = torch.accelerator.memory_stats(device)["allocated_bytes.all.peak"]
self.free_memory, self.total_memory = current_platform.mem_get_info(device)
self.cuda_memory = self.total_memory - self.free_memory       # everything in use, by anyone
self.torch_memory = torch.accelerator.memory_reserved(device) # what PyTorch reserved
self.non_torch_memory = self.cuda_memory - self.torch_memory  # category 3

The subtlety that trips people up: non_torch_memory (category 3) is computed as “total in-use minus what PyTorch reserved.” It captures the CUDA context, NCCL buffers, and some attention backends’ scratch — memory that gpu_memory_utilization must pay for but that no PyTorch counter sees. This is why a naive “weights + KV” mental model always under-counts, and why two engines tuned to sum to 1.0 can still OOM: each has its own invisible category-3 tax.

4. determine_available_memory: profile, then subtract

Now the budget gets spent. determine_available_memory() runs after weights are loaded, and it does exactly one empirical thing: a dummy forward pass at the largest batch the engine will ever see, wrapped in the profiler.

# vllm/v1/worker/gpu_worker.py:384-399
with memory_profiling(
    self.init_snapshot,
    weights_memory=int(self.model_runner.model_memory_usage),
) as profile_result:
    self.model_runner.profile_run()          # forward pass on dummy max-token batch
    ...
    if not self.model_config.enforce_eager and not current_platform.is_rocm():
        cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()

profile_run() shoves a synthetic batch of max_num_batched_tokens through the model so the peak activation memory is observed, not guessed. The profiler then assembles the non-KV total (gpu_worker.py:405):

# vllm/v1/worker/gpu_worker.py:405-409
profile_result.non_kv_cache_memory = (
    profile_result.non_torch_increase      # category 3 growth (NCCL, buffers)
    + profile_result.torch_peak_increase   # category 2 activation peak
    + profile_result.weights_memory        # category 2 weights
)

And finally the line that decides your KV cache (gpu_worker.py:437):

# vllm/v1/worker/gpu_worker.py:437-441
self.available_kv_cache_memory_bytes = (
    self.requested_memory                  # total × util  (§2)
    - profile_result.non_kv_cache_memory   # weights + activation + non-torch  (§4)
    - cudagraph_memory_estimate_applied    # CUDA-graph capture pool
)

That is the whole story. gpu_memory_utilization sets the ceiling (requested_memory); weights, the measured activation peak, the non-torch tax, and the CUDA-graph pool are subtracted; and the KV cache gets whatever is left. vLLM then logs that remainder — the “Available KV cache memory: X GiB” line at gpu_worker.py:457 — which is the number you should actually watch, not 0.92.

One ordering note that matters in practice: because the activation peak is measured by a real forward pass, anything already resident on the GPU when vLLM profiles is correctly accounted as category 1 or 3. This is why frameworks that colocate vLLM with training (verl’s hybrid engine, for instance) deliberately create the vLLM engine last — so its KV estimate reflects the FSDP allocations that already happened, rather than a phantom-empty GPU.

5. When it doesn’t fit: the CUDAGraph overflow message

If CUDA graphs are enabled (the default — enforce_eager=False), their capture pool is a fourth subtraction, and it’s the one most likely to push you negative. vLLM has a dedicated diagnostic for it (gpu_worker.py:641-669):

# vllm/v1/worker/gpu_worker.py:641-669 (condensed)
non_kv_cache_memory = (
    self.model_runner.model_memory_usage    # weights
    + self.peak_activation_memory           # activation peak
    + self.non_torch_memory                 # category 3
    + cuda_graph_memory_bytes               # the graph pool
)
...
f"Actual usage is {weights} GiB for weight, {peak_activation} GiB for peak "
f"activation, {non_torch} GiB for non-torch memory, and {cuda_graph} GiB "
f"for CUDAGraph memory. Replace gpu_memory_utilization config with "
f"`--kv-cache-memory=...`"

That message is the single most useful line vLLM prints when you’re memory-tuning: it itemizes all four subtractions in GiB, so you can see which term ate the budget rather than guessing. If weights dominates, you need more parallelism (TP/EP) or quantization; if peak activation dominates, lower max_num_batched_tokens; if there’s nothing left for KV, lower max_model_len or accept lower concurrency.

Putting it together: the 235B reward server

Here is the real example that sent me into this code, with the numbers from vLLM’s own startup log:

Term	Value (per GPU)	Source
total_memory (H200)	~140.4 GiB	mem_get_info
× gpu_memory_utilization = 0.92	129.2 GiB	requested_memory, §2
− model weights (Qwen3-235B, TP4)	109.6 GiB	weights_memory
− activation peak + non-torch	~12.7 GiB	profile run, §4
− CUDA graph pool	1.4 GiB	profile_cudagraph_memory
= Available KV cache	≈ 5.5 GiB	gpu_worker.py:437

0.92 sounds like “almost the whole GPU for serving,” but the 235B MoE weights are 110 GiB of it, so the KV cache is a 5.5 GiB sliver — about 122k tokens, or ~60 concurrent 2,000-token prompts. That single fact explained everything downstream: the model was weight-bound, KV-cache tuning was nearly useless, and the moment I colocated it (dropping util to 0.5 to share the GPU with a rollout engine) the KV cache shrank to roughly a quarter — which is precisely why the colocated reward throughput collapsed. None of that is visible from 0.92; all of it is visible from Available KV cache memory: 5.5 GiB.

The mechanical takeaways, straight from the four files:

• gpu_memory_utilization is a ceiling on total, not a KV-cache size and not a slice of free memory. Read it as “how much of the card this one engine may occupy in total.”
• KV cache is a residual. It is whatever survives total × util − weights − activation − non-torch − cudagraph. Big weights → tiny KV, at any utilization.
• The number to watch is the logged Available KV cache memory, not the knob. When that goes negative you get the §5 message, which tells you exactly which term to attack.
• It’s per-instance and uncoordinated. Colocating engines means you partition the card; the sum of their util values (plus everyone’s invisible category-3 tax) must clear 1.0 with room to spare.

Where to read next

• vllm/v1/worker/gpu_worker.py:determine_available_memory — the function in full; the subtraction and all its logging.
• vllm/utils/mem_utils.py:memory_profiling — the three-category model with vLLM’s own worked numeric example.
• vllm/v1/core/kv_cache_utils.py — where available_kv_cache_memory_bytes becomes a block/token count (the “GPU KV cache size: N tokens” and “Maximum concurrency” logs).
• vLLM conserving-memory docs — the user-facing knobs (gpu_memory_utilization, max_num_batched_tokens, max_model_len, quantization) mapped back to the terms above.