vLLM v1 Deep Dive: How LLM(...) and the Server Boot and Generate

Introduction

vLLM v1 is the rewrite of vLLM that landed in 2024 and is now the default. The public API still looks like:

from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", gpu_memory_utilization=0.9)
out = llm.generate(["What's the capital of France?"], SamplingParams(max_tokens=64))
print(out[0].outputs[0].text)

…but the internals are a different beast from v0. There’s now a clear process boundary between the user-facing LLM and the GPU-driving EngineCore, with ZMQ between them. There’s a strict layer hierarchy. There’s parallel sampling fan-out, prefix caching, KV-cache auto-fitting, CUDA graph capture, and per-request output streaming — most of it happening across two processes.

This post traces what actually happens when you call LLM(...) and then llm.generate(...), with permalinks pinned to commit 6d0976970 of the vLLM repo.

Parts 1–3 follow the offline LLM path — the synchronous “I have a script and some prompts, run them” usage — because it’s the simplest way to expose the architecture. Part 5 then traces the online server (vllm serve / AsyncLLM): its startup, how it handles a request, and where CUDA graphs are captured — showing that both paths converge on the same EngineCore.

Part 1 — The Six-Layer Architecture

vLLM v1 stacks six layers from the user-facing API down to the GPU:

┌─────────────────────────────────────────────────────────────────────────────┐
│ LAYER 1: LLM (vllm/entrypoints/llm.py)                          MAIN process│
│   User-facing API: LLM(model=...).generate(prompts, sampling_params)        │
│   Owns: tokenizer, request_counter, llm_engine                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │ holds
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ LAYER 2: LLMEngine (vllm/v1/engine/llm_engine.py)               MAIN process│
│   Synchronous batch driver around EngineCore.                               │
│   Owns: input_processor, output_processor, engine_core (an EngineCoreClient)│
└─────────────────────────────────────────────────────────────────────────────┘
                                    │ holds
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ LAYER 3: EngineCoreClient (vllm/v1/engine/core_client.py)       MAIN process│
│   Abstract IPC client. Three concrete subclasses:                           │
│     • InprocClient — runs EngineCore in *this* process (debug only)         │
│     • SyncMPClient — for offline LLM (multiprocess, blocking)               │
│     • AsyncMPClient — for AsyncLLM / OpenAI server                          │
└─────────────────────────────────────────────────────────────────────────────┘
                              ┌─────┘
                              │ owns the spawn lifecycle and the input/output
                              │ ZMQ sockets; everything below runs in a
                              │ DIFFERENT process spawned via mp "spawn" context
                              ▼
═══════════════════════════════ PROCESS BOUNDARY ═══════════════════════════════
                              ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ LAYER 4: EngineCoreProc (vllm/v1/engine/core.py)               EngineCore   │
│   Wraps EngineCore with a busy loop polling the input zmq socket.           │
│   Owns: input_queue, output_queue, scheduler, model_executor                │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │ holds
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ LAYER 5: Executor (vllm/v1/executor/...)                       EngineCore   │
│   Abstracts TP/PP fan-out:                                                  │
│     • UniprocExecutor — TP=1 PP=1 (no extra subprocesses)                   │
│     • MultiprocExecutor — spawns one Worker subprocess per TP rank          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │ holds
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ LAYER 6: Worker / GPUModelRunner (vllm/v1/worker/gpu_worker.py)             │
│   The actual GPU rank. Owns the model weights, KV-cache pool,               │
│   InputBatch, Sampler, LogitsProcessors, CUDA graphs.                       │
└─────────────────────────────────────────────────────────────────────────────┘

The key thing this picture buys you: everything above the dashed line lives in your Python script’s process. Everything below lives in a separate spawned process. The boundary is ZMQ. Layer 3 is the abstraction that decides whether to fork or stay in-process.

🧠 Why two processes? Originally vLLM was single-process. The split was added in v1 because serving (where many requests overlap) wants the model loop running on its own thread of control, free of GIL contention with HTTP / streaming code. The downside for the offline path is that every llm.generate(...) pays a per-call IPC cost — fine when you batch a lot of work into one call, not great when you make many small calls.

For TP=1 (single GPU) with LLM(...), exactly one extra subprocess is spawned: the EngineCoreProc. The model weights, KV cache, sampler, and any custom LogitsProcessors live inside that subprocess. With TP > 1, the MultiprocExecutor would fan out further into TP-many Worker subprocesses, each holding a model shard.

Layer permalinks

Layer	Class	File:line
1	LLM	vllm/entrypoints/llm.py:212
2	LLMEngine	vllm/v1/engine/llm_engine.py:50
3	EngineCoreClient	vllm/v1/engine/core_client.py:69
3	SyncMPClient	core_client.py:716
3	AsyncMPClient	core_client.py:887
4	EngineCore	vllm/v1/engine/core.py:92
4	EngineCoreProc	vllm/v1/engine/core.py:802
5	UniprocExecutor	uniproc_executor.py
5	MultiprocExecutor	multiproc_executor.py
6	Worker	vllm/v1/worker/gpu_worker.py

Part 2 — Initialization: From LLM(...) to “ready”

What actually happens when you write llm = LLM("Llama-3.1-8B-Instruct")?

Wall time on a typical setup (H100 80GB, 1.7B-parameter model, gpu_memory_utilization=0.72, enforce_eager=False): about 14-15 seconds end-to-end, dominated by model weight loading + CUDA graph capture. Bigger models, slower NFS, or enforce_eager=False (which triggers torch.compile) push it higher.

Let’s walk it phase by phase.

Phase 1 — Caller-process construction

LLM.__init__ parses arguments into EngineArgs and calls LLMEngine.from_engine_args(...):

self.llm_engine = LLMEngine.from_engine_args(...)

Inside LLMEngine.__init__ the caller-process construction does three notable things, all before any subprocess is spawned:

# vllm/v1/engine/llm_engine.py:93,96,104
self.input_processor = InputProcessor(self.vllm_config, renderer)        # main process
self.output_processor = OutputProcessor(...)                             # main process
self.engine_core = EngineCoreClient.make_client(                         # spawns!
    multiprocess_mode=multiprocess_mode,
    asyncio_mode=False,
    vllm_config=vllm_config,
    executor_class=executor_class,
    log_stats=log_stats,
)

InputProcessor owns the tokenizer and validates incoming prompts. It stays in the caller process; every LLMEngine.add_request will hit it.

OutputProcessor also stays in the caller process. It owns one RequestState per active request, with a LogprobsProcessor and IncrementalDetokenizer attached. Per-request streaming output construction happens here.

The third line is the magic one: constructing engine_core is what triggers the subprocess spawn.

Phase 2 — Client dispatch

EngineCoreClient.make_client is a tiny dispatcher:

# vllm/v1/engine/core_client.py:88-103
if asyncio_mode and not multiprocess_mode:
    raise NotImplementedError(...)
if multiprocess_mode and asyncio_mode:
    return AsyncMPClient(...)        # OpenAI server / AsyncLLM
if multiprocess_mode and not asyncio_mode:
    return SyncMPClient(...)         # offline LLM — our path
return InprocClient(...)             # debug / dev only — same process

For the offline LLM(...) path this returns a SyncMPClient, which calls MPClient.__init__ as super().__init__(). That constructor:

1. Sets up two ZMQ sockets in the main process (core_client.py:511-533):
   • input_socket — caller pushes EngineCoreRequests here
   • output_socket — caller pulls EngineCoreOutputs from here
2. Calls launch_core_engines(...) which is where the subprocess is born.
3. Waits for ready-handshake on the input socket (core_client.py:577-595) — every spawned EngineCore must send back an EngineCoreReadyResponse before the client returns.

Phase 3 — The Spawn Site

This is the bit worth knowing precisely. launch_core_engines constructs a CoreEngineProcManager, whose constructor at utils.py:101-135 reads:

# vllm/v1/engine/utils.py:101-133
def __init__(self, ..., vllm_config, executor_class, log_stats, ...):
    context = get_mp_context()           # ← returns spawn context
    common_kwargs = {
        "vllm_config": vllm_config,
        "executor_class": executor_class,
        ...
    }
    from vllm.v1.engine.core import EngineCoreProc

    self.processes: list[BaseProcess] = []
    for index in range(local_engine_count):
        ...
        self.processes.append(
            context.Process(                                       # ← THIS LINE
                target=EngineCoreProc.run_engine_core,             # ← target fn
                name="EngineCore",
                kwargs=common_kwargs | {"dp_rank": ..., ...},
            )
        )

That context.Process(...) call at utils.py:128-133 is the spawn site — the line that creates the EngineCore subprocess.

get_mp_context() is at vllm/utils/system_utils.py:168 and returns multiprocessing.get_context("spawn") by default (controlled by env var VLLM_WORKER_MULTIPROC_METHOD). Its preceding helper _maybe_force_spawn forces spawn under conditions like CUDA-already-initialized — so for any GPU inference path, spawn is effectively always used.

🔥 Spawn semantics matter Spawned children re-import everything from clean Python state. Runtime monkey-patches you’ve made in the parent process do NOT propagate to the child. The child does inherit the parent’s environment variables.

The canonical place to inject behavior into a spawned child is a sitecustomize.py on PYTHONPATH — Python’s site module loads it at every interpreter startup, before the user’s imports. Combined with an env-var gate (which the child inherits from the parent), this gives you a clean way to install a per-process patch only when you opt in.

If you’ve ever wondered why some patches “work in test but not in production” — it’s often this. You patched the parent; vLLM ran in the spawned child.

Phase 4 — In the spawned child: run_engine_core

The new process’s entry point is EngineCoreProc.run_engine_core at core.py:1060. This is a static function, NOT a method. That’s deliberate: spawn semantics make unpickling self-references awkward, so vLLM passes only kwargs across the boundary and reconstructs the object in the child.

# vllm/v1/engine/core.py:1060
def run_engine_core(*args, dp_rank: int = 0, local_dp_rank: int = 0, **kwargs):
    # Ensure HF transformer config can serialize after spawning
    maybe_register_config_serialize_by_value()
    ...
    set_process_title("EngineCore")
    ...
    engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
    ...
    engine_core.run_busy_loop()  # never returns until shutdown

Phase 5 — EngineCoreProc.__init__ → EngineCore.__init__

EngineCoreProc.__init__ sets up input_queue / output_queue (in-process Python queues feeding the busy loop), performs the ZMQ handshake with the parent, then calls super().__init__(...) — which is the heavy EngineCore.__init__.

EngineCore.__init__ does the actual model bring-up. The interesting parts:

# vllm/v1/engine/core.py (abbreviated)
class EngineCore:
    def __init__(self, vllm_config, executor_class, log_stats, ...):
        load_general_plugins()                                   # 1) plugins

        self.vllm_config = vllm_config
        self.log_stats = log_stats

        # 2) Build executor (which loads the model)
        self.model_executor = executor_class(vllm_config)        # core.py:116

        # 3) Profile peak memory + allocate KV cache pool
        kv_cache_config = self._initialize_kv_caches(vllm_config)  # core.py:131

        # 4) Build scheduler
        Scheduler = vllm_config.scheduler_config.get_scheduler_cls()
        self.scheduler = Scheduler(...)                          # core.py:140

Each step is doing real work:

Step 2 (the executor) is what actually loads the model. For TP=1 it’s UniprocExecutor — no further spawning. For TP > 1 it’s MultiprocExecutor which spawns N Worker subprocesses, one per TP rank. The Executor wraps GPUModelRunner which holds the actual nn.Module, KV cache pages, InputBatch, Sampler, LogitsProcessors, and CUDA graphs.

Step 3 is the slow one — let’s zoom in.

Phase 5b — KV cache profiling and allocation

_initialize_kv_caches is the most architecturally interesting part of init:

# vllm/v1/engine/core.py:231-282 (abbreviated)
def _initialize_kv_caches(self, vllm_config):
    kv_cache_specs = self.model_executor.get_kv_cache_specs()        # 1
    available_gpu_memory = self.model_executor.determine_available_memory()  # 2
    kv_cache_configs = get_kv_cache_configs(...)                     # 3
    self.model_executor.initialize_from_config(kv_cache_configs)     # 4

1. get_kv_cache_specs — query each layer’s KV-cache shape requirements (head dim, num heads, dtype, block size). Different layer types have different needs (attention vs Mamba vs grouped query etc.).
2. determine_available_memory — runs a profiling forward pass on a fake max-batch input to measure peak GPU memory consumed by activations and intermediate buffers. Subtracts from gpu_memory_utilization × free_GPU_memory to determine the budget for KV cache. This is why you sometimes see brief memory spikes during init: that’s the profile.
3. get_kv_cache_configs — given the budget and the per-layer specs, compute optimal block sizes and counts. May reduce max_model_len if the requested length doesn’t fit; the reduced value is propagated back to the caller in the ready handshake.
4. initialize_from_config — actually allocate the KV cache pool, run a warmup forward, and capture CUDA graphs. The Capturing CUDA graphs ... 100% [N/N] lines you see in vLLM startup logs are step 4.

After this returns, EngineCore.__init__ continues to step 4 (build the Scheduler) and logs the now-famous line:

init engine (profile, create kv cache, warmup model) took 14.62 s (compilation: 3.10 s)

Phase 6 — Ready handshake back to caller

After EngineCore.__init__ returns, EngineCoreProc sends an EngineCoreReadyResponse (msgspec-encoded) back through input_socket to the parent’s MPClient. The parent has been polling at core_client.py:579-595 with VLLM_ENGINE_READY_TIMEOUT_S timeout (default 600s).

When the response arrives, MPClient._apply_ready_response syncs back any auto-adjusted config — notably a reduced max_model_len if KV-cache auto-fitting decided the requested length wouldn’t fit. This is worth knowing: if you ask for max_model_len=32768 but the KV pool only fits 20480 tokens, the engine quietly reduces it and updates the caller’s config.

Phase 7 — Final wiring and return

Before returning, MPClient.__init__ at core_client.py:641-665 spawns a daemon thread MPClientEngineMonitor that polls subprocess liveness. If the EngineCore subprocess dies unexpectedly, the thread sets resources.engine_dead=True, causing all subsequent add_request / get_output calls in the parent to raise EngineDeadError. This is the mechanism that gives you a clean error rather than a hang when the GPU side crashes.

MPClient.__init__ returns. LLMEngine.__init__ continues, runs get_supported_tasks() (round-trips to EngineCore via the now-live ZMQ pipes), and finalizes its state. Control returns to LLM.__init__, which sets up model_config, engine_class, renderer, input_processor shortcuts.

LLM(...) returns. The subprocess is alive, idle in run_busy_loop, waiting for the first add_request.

Wall-clock breakdown

Approximate timing for a 1.7B-parameter model on a H100 80GB:

phase	est. ms	what’s happening
Caller init + spawn fork-exec	~500	Python interpreter startup, vllm imports
Plugin loading + executor_class(...)	~3000	model weight load, torch.compile cache hit
determine_available_memory (peak profile)	~1000	one forward pass on max batch
initialize_from_config (KV alloc + CUDA graphs)	~4000	piecewise + full CUDA graphs captured
Ready-handshake + scheduler init + warmup	~6000	torch.compile transform, cudagraph profiling
Total	~14,500	matches “init engine … took ~14.5 s”

The init cost is one-time per LLM(...) instance. Across all subsequent llm.generate(...) calls, the subprocess stays alive. It only dies when your LLM object is garbage-collected (which triggers MPClient.shutdown).

Part 3 — Runtime: One llm.generate(...) Call

After init, calling llm.generate(prompts, sampling_params) is the hot path. Let’s trace it.

The high-level flow is: caller adds N requests → engine processes them → caller collects N RequestOutputs:

caller process                                EngineCore subprocess
─────────────────                             ──────────────────────
LLM.generate(prompts, params)
 ↓
LLM._add_completion_requests
 ↓ (loop over N)
LLM._add_request
 ↓
LLMEngine.add_request                         (via ZMQ, send_multipart)
 ├─ InputProcessor.process_inputs    ──────▶  EngineCore.add_request
 ├─ OutputProcessor.add_request                  ↓
 └─ engine_core.add_request                   self.scheduler.waiting.append
                                                 ↓
LLMEngine._run_engine                         busy loop: schedule → forward → sample
 └─ while has_unfinished_requests:               ↓
   LLMEngine.step                             EngineCoreOutputs ──▶ output_socket
    └─ engine_core.get_output       ◀──────  (via ZMQ, recv_multipart)
       └─ Connection.wait + zmq.recv
   OutputProcessor.process_outputs
    ├─ detokenizer.update
    ├─ logprobs_processor.update_from_output
    └─ make_request_output

Step 1 — Caller-side: rendering and per-Request fan-out

Entry: LLM.generate → LLM._run_completion → LLM._add_completion_requests.

_add_completion_requests calls _render_and_add_requests, which loops over the N prompts and calls _add_request for each, which calls LLMEngine.add_request.

Inside LLMEngine.add_request:

# vllm/v1/engine/llm_engine.py:209-285 (abbreviated)
def add_request(self, request_id, prompt, params, ...):
    # 1) Tokenize (no-op if caller passed pre-tokenized prompt)
    request = self.input_processor.process_inputs(...)             # llm_engine.py:241

    # 2) Register output state (RequestState, LogprobsProcessor, IncrementalDetokenizer)
    n = params.n if isinstance(params, SamplingParams) else 1
    if n == 1:
        self.output_processor.add_request(request, prompt_text, None, 0)
        self.engine_core.add_request(request)                      # zmq send to subprocess
        return req_id

    # 3) Parallel sampling (n>1): fan out to N child requests
    parent_req = ParentRequest(request)
    for idx in range(n):
        request_id, child_params = parent_req.get_child_info(idx)
        child_request = request if idx == n - 1 else copy(request)
        child_request.request_id = request_id
        child_request.sampling_params = child_params
        self.output_processor.add_request(child_request, prompt_text, parent_req, idx)
        self.engine_core.add_request(child_request)                # one zmq send per child!
    return req_id

Two things are worth noting:

Parallel sampling is just N independent Requests. SamplingParams(n=k) fans out at the LLMEngine.add_request layer into K child requests, each with sampling_params.n=1. The Scheduler, KVCacheManager, Sampler, and OutputProcessor downstream see K completely independent Requests. There is no MultiRowReq primitive in v1: one Request always means one InputBatch row. Prefix caching dedupes the prompt KV across the children, but the per-Request orchestration is still K-fold.

Each engine_core.add_request is a ZMQ round-trip. Internally it’s EngineCoreClient.add_request → _send_input → zmq.send_multipart(...). Per-call cost is small (microseconds), but it multiplies by N.

Step 2 — In the subprocess: input dispatch

EngineCoreProc.run_busy_loop in the spawned child polls input_socket, deserializes via msgspec, and routes by EngineCoreRequestType:

• ADD → EngineCore.add_request
• ABORT, UTILITY, etc. handled separately.

EngineCore.add_request builds an internal Request, runs update_block_hashes (computes prefix-cache keys for the prompt), and pushes to self.scheduler.waiting.

Step 3 — Scheduler iteration

Scheduler.schedule runs once per engine iteration, immediately before model_executor.execute_model. Its output is the exact batch that the workers execute.

The scheduler does not directly choose “N requests”

This is the most important point: there is no code equivalent to batch_size = 18. The request count is an output of greedy admission under several constraints.

The two main configured limits are initialized here:

# vllm/v1/core/sched/scheduler.py:104-110
self.max_num_running_reqs = self.scheduler_config.max_num_seqs
self.max_num_scheduled_tokens = (
    self.scheduler_config.max_num_scheduled_tokens
    or self.scheduler_config.max_num_batched_tokens
)

max_num_batched_tokens limits tokens processed in one iteration, while max_num_seqs limits how many requests may be admitted into self.running.

At the beginning of every step, vLLM creates a fresh token budget:

# vllm/v1/core/sched/scheduler.py:365-370
num_scheduled_tokens = {}
token_budget = self.max_num_scheduled_tokens

The resulting batch satisfies roughly:

[ \text{scheduled request count} = \left|\texttt{num_scheduled_tokens}\right| ]

[ \sum_r \texttt{num_scheduled_tokens}[r] \le \texttt{max_num_scheduled_tokens} ]

but the exact count also depends on KV-cache space, prefix-cache hits, request ordering, LoRA/encoder constraints, and whether prefill chunking is enabled. This is a greedy FCFS/priority scheduler, not a bin-packing optimizer that searches for the maximum-cardinality batch.

One unified “token deficit” model

The scheduler explicitly says there is no fundamental prefill/decode distinction. Each Request has tokens that exist and tokens that have already been computed; scheduling tries to close that deficit (source):

num_new_tokens = (
    request.num_tokens_with_spec
    + request.num_output_placeholders
    - request.num_computed_tokens
)

For ordinary decode, the deficit is normally one token. For an uncached prefill, it can be the entire prompt. For a prefix-cache hit, it is only the uncached suffix.

Pass 1: keep existing running requests moving

vLLM schedules self.running first (source). For each running Request it:

1. computes the token deficit,
2. caps it by long_prefill_token_threshold,
3. caps it by the remaining token_budget,
4. caps it so the sequence cannot exceed max_model_len.

It then asks the KV-cache manager to allocate slots (source). If allocation fails, it preempts requests until either space becomes available or the current request itself has been evicted.

# vllm/v1/core/sched/scheduler.py:512-518
scheduled_running_reqs.append(request)
num_scheduled_tokens[request_id] = num_new_tokens
token_budget -= num_new_tokens

That subtraction is the mechanism that indirectly limits how many additional requests can enter this iteration.

Pass 2: admit waiting requests into the remaining budget

Only after processing running requests does vLLM inspect self.waiting (source):

while (self.waiting or self.skipped_waiting) and token_budget > 0:
    if len(self.running) == self.max_num_running_reqs:
        break
    request = request_queue.peek_request()

This is where the sequence-count limit matters: max_num_seqs prevents more requests from being promoted into running. It is not necessarily equal to the number executed in the current step—some already-running requests may be temporarily unschedulable.

For a new Request, prefix-cache matching happens before the scheduler decides num_new_tokens.

When a Request is constructed, vLLM hashes every complete token block (Request.update_block_hashes, get_request_block_hasher). Each block hash includes the previous block’s hash, the current block’s token IDs, and relevant extra keys such as LoRA, multimodal inputs, prompt embeddings, or cache salt (hash_block_tokens). The parent hash makes this a chain: a block matches only when the entire prefix leading to it also matches.

During scheduling, vLLM calls kv_cache_manager.get_computed_blocks(request). That method limits the maximum hit to request.num_tokens - 1, because even a fully cached prompt must recompute its final position to produce logits (source):

max_cache_hit_length = request.num_tokens - 1
computed_blocks, num_new_computed_tokens = (
    self.coordinator.find_longest_cache_hit(
        request.block_hashes,
        max_cache_hit_length,
    )
)

For an ordinary full-attention model, the coordinator converts matched blocks back into a token count (source). The actual longest-prefix scan is:

# vllm/v1/core/single_type_kv_cache_manager.py:446-457
for block_hash in block_hashes[:max_num_blocks]:
    cached_block = block_pool.get_cached_block(block_hash, kv_cache_group_ids)
    if cached_block:
        computed_blocks.append(cached_block)
    else:
        break

See the full implementation in FullAttentionManager.find_longest_cache_hit. It stops at the first miss because later chained hashes cannot be valid if an earlier prefix block is absent.

The lookup itself is a hash-table query in BlockPool.get_cached_block. Newly completed full blocks enter that table through BlockPool.cache_full_blocks.

Back in schedule, local and externally loaded cache hits are added together, then subtracted from the Request’s token count:

# vllm/v1/core/sched/scheduler.py:639-689
num_computed_tokens = (
    num_new_local_computed_tokens
    + num_external_computed_tokens
)
num_new_tokens = request.num_tokens - num_computed_tokens

if 0 < long_prefill_token_threshold < num_new_tokens:
    num_new_tokens = long_prefill_token_threshold

if not enable_chunked_prefill and num_new_tokens > token_budget:
    break

num_new_tokens = min(num_new_tokens, token_budget)

For example, if a 1,000-token prompt has 800 tokens represented by matched full blocks, this iteration schedules only 200 tokens. Matching is block granular, so a final partial block—and sometimes a complete final block—must be recomputed.

The break is significant. With chunked prefill disabled, a prompt that does not fit in the remaining budget is not partially scheduled, and vLLM stops admitting more waiting requests in that pass. With chunked prefill enabled, min(..., token_budget) lets it consume the remaining budget as a partial prefill.

The request still needs enough physical KV capacity. vLLM calls allocate_slots; if it returns None, admission stops even if the numerical token and sequence budgets have room.

Once admitted, the Request moves to running, its token allocation is recorded, and the budget decreases (source):

self.running.append(request)
num_scheduled_tokens[request_id] = num_new_tokens
token_budget -= num_new_tokens
request.status = RequestStatus.RUNNING

A concrete production-shaped example

Suppose:

max_num_batched_tokens = 16,384
max_num_seqs           = 1,024
chunked prefill        = disabled
running requests       = 0

After prefix-cache lookup, 18 waiting requests require a total of 16,117 prompt tokens. The scheduler admits all 18:

remaining token budget = 16,384 - 16,117 = 267

If the next waiting prompt needs 900 uncached tokens, it does not fit. Because chunked prefill is disabled, the scheduler breaks instead of scheduling 267 tokens from it. The forward batch therefore contains 18 requests, not because 18 was selected explicitly, but because the nineteenth request failed the next greedy admission test.

On a later mixed iteration, the same algorithm might first schedule 18 running decode requests at one token each, then use the remaining 16,366-token budget for new prefills. This is how a trace annotation such as execute_context_14(13747)_generation_18(18) arises: one scheduler output contains 14 context requests/13,747 context tokens plus 18 generation requests/18 generation tokens.

The final answer is num_scheduled_tokens

After both passes, vLLM verifies the invariants (source):

total_num_scheduled_tokens = sum(num_scheduled_tokens.values())
assert total_num_scheduled_tokens <= self.max_num_scheduled_tokens
assert len(self.running) <= self.max_num_running_reqs

It packages the per-request mapping into SchedulerOutput, then EngineCore.step passes that object directly to the model executor.

One subtle bookkeeping detail: after constructing the output, the scheduler advances each Request’s num_computed_tokens by the scheduled amount (source). The GPU has not completed yet; this is scheduler-side planned state. If speculative tokens are rejected, update_from_output corrects the count later.

Two consequences worth remembering:

Block allocation is per-Request. allocate_slots consults the prefix cache (hash lookup on the prompt block hashes computed earlier) to dedupe KV blocks where possible.

Preemption is a thing. When the engine fills up, the scheduler will evict lower-priority requests to make room. Their KV is freed; if they’re rescheduled later they prefill again from the deduped prompt blocks (if still cached).

Step 4 — Forward + sampling

GPUModelRunner.execute_model runs the forward pass on the assembled batch. Input prep is at _prepare_inputs which flattens scheduled requests into the (total_tokens, ...) GPU input tensor — one InputBatch row per Request, with num_scheduled_tokens tokens per row (typically 1 in decode, len(prompt) in prefill).

The forward pass produces (num_rows, vocab) logits. Sampler.forward runs:

# vllm/v1/sample/sampler.py:68-143 (abbreviated)
def forward(self, logits, sampling_metadata, ...):
    # 1) raw logprobs (if requested)
    if num_logprobs is not None:
        if logprobs_mode == "raw_logprobs":
            raw_logprobs = self.compute_logprobs(logits)

    # 2) Apply non-argmax-invariant LogitsProcessors (incl. user-provided)
    logits = self.apply_logits_processors(logits, ...)

    # 3) Sample (greedy / temperature / top-k / top-p)
    sampled, processed_logprobs = self.sample(logits, sampling_metadata)

    # 4) Gather top-K logprobs for output
    logprobs_tensors = self.gather_logprobs(raw_logprobs, num_logprobs, ...)

    return SamplerOutput(sampled_token_ids=sampled.unsqueeze(-1), ...)

The order is important: logprobs are computed before user LogitsProcessors run by default. If you’re masking the logit space (e.g., constrained decoding), you almost certainly want to set logprobs_mode="processed_logprobs" on LLM(...) — otherwise the reported logprobs reflect the unmasked distribution and downstream top-K may return tokens your mask would have forbidden.

Step 4b — Custom LogitsProcessors

vLLM v1’s LogitsProcessor base lives at vllm/v1/sample/logits_processor/interface.py. It exposes two methods:

# vllm/v1/sample/logits_processor/interface.py
class LogitsProcessor(ABC):
    @abstractmethod
    def update_state(self, batch_update: "BatchUpdate | None") -> None:
        """Called when batch composition changes (rows added/removed/moved)."""

    @abstractmethod
    def apply(self, logits: torch.Tensor) -> torch.Tensor:
        """Called every forward pass. logits is (batch_size, vocab). May modify in-place."""

For per-request logic, vLLM provides AdapterLogitsProcessor as a convenience subclass:

class AdapterLogitsProcessor(LogitsProcessor):
    def apply(self, logits):
        for req_idx, req_lp in self.req_info.items():
            req_logits = logits[req_idx]
            new_logits = req_lp(req_logits)
            if new_logits is not req_logits:
                logits[req_idx] = new_logits
        return logits

You implement new_req_logits_processor(params) returning a (output_ids, logits_row) -> logits_row callable. The Adapter pattern dispatches it per-row internally.

LogitsProcessors are wired via LLM(..., logits_processors=[CustomLogitsProcessorClass]). The class is instantiated once per Worker (in the EngineCore subprocess); per-request state lives in the instance and gets populated via update_state callbacks when new requests join the batch.

Step 4c — Mixed batches: prefill + decode in one forward, and sample_tokens

A question that confuses everyone reading a vLLM v1 trace: you see an iteration annotated execute_context_1(4)_generation_5(5) and ask — does a single forward really run prefill and decode at the same time? Yes. vLLM v1 has no separate “prefill engine” and “decode engine”; continuous batching folds both into one execute_model call. The annotation is literally telling you the mix: 1 request prefilling (4 prompt tokens) + 5 requests decoding (5 tokens, 1 each) in this forward.

How the two phases get flattened into one batch. The scheduler is phase-agnostic — num_scheduled_tokens[req_id] is just “how many tokens this request contributes this step” (its remaining prompt chunk in prefill, 1 in decode). _prepare_inputs concatenates every scheduled request’s tokens into one 1-D input_ids of length total_num_scheduled_tokens, and builds query_start_loc — the cumulative-sum boundaries (cu_seqlens) that mark where each request’s slice starts:

# vllm/v1/worker/gpu_model_runner.py:1888-1894 (abbreviated)
self.query_start_loc.np[0] = 0
self.query_start_loc.np[1 : num_reqs + 1] = cu_num_tokens   # prefix sum of per-req token counts
# e.g. prefill A=4 tok, then 5 decodes of 1 tok ->
#   input_ids  : [a0 a1 a2 a3 | b0 | c0 | d0 | e0 | f0]   (length 9)
#   query_start_loc: [0, 4, 5, 6, 7, 8, 9]

From the GEMMs’ point of view there is no “prefill vs decode” — the linear and MoE layers just matmul [total_tokens, hidden]; the M dimension is the total token count, a mix of the 4 prefill tokens and 5 decode tokens. The only layer that needs to know the boundaries is attention, which runs in varlen mode: query_start_loc + seq_lens + the paged block tables let one FlashAttention/FlashInfer kernel give the prefill request a causal mask over its 4 new tokens (attending to its cached prefix) and give each decode request a single query attending to all of its cached KV — in the same kernel launch. That is the whole trick: flatten everything, let varlen attention respect the per-request boundaries, and the rest of the network is phase-blind.

Where decode and prefill actually diverge: the LM head. A prefill request that contributed 4 tokens does not need 4 rows of logits — only its last position predicts the next token. So after the transformer stack, vLLM gathers just the final row of each request before the (expensive) vocab projection:

# vllm/v1/worker/gpu_model_runner.py:2037 — one logits row per request
logits_indices = query_start_loc[1:] - 1     # [3, 4, 5, 6, 7, 8]  (last token of each slice)

# vllm/v1/worker/gpu_model_runner.py:4070-4071 — slice THEN project
sample_hidden_states = hidden_states[logits_indices]      # [num_reqs, hidden], not [total_tokens, hidden]
logits = self.model.compute_logits(sample_hidden_states)  # LM head only on num_reqs rows

So no matter how prefill-heavy the forward is, compute_logits and the sampler always see exactly num_reqs rows (here 6) — one candidate position per request.

Why execute_model and sample_tokens are split. In v1’s V2 runner the forward and the sampling are two separate worker entry points (gpu_worker.py:748 — this is the gpu_worker.py(774): sample_tokens frame you see in a 0.22 trace; the line moved a little across patches):

# vllm/v1/worker/gpu_worker.py:748-751, 813-814
def sample_tokens(self, grammar_output):
    return self.model_runner.sample_tokens(grammar_output)   # -> gpu_model_runner.py:4124

...
with self.annotate_profile(scheduler_output):                # <- emits "execute_context_..._generation_..."
    output = self.model_runner.execute_model(scheduler_output)

execute_model launches the forward and compute_logits, stashes the result in self.execute_model_state, and returns None without sampling. sample_tokens then runs the Sampler (greedy / top-k / top-p) on those num_reqs logit rows and produces the ModelRunnerOutput. The split exists for async scheduling: once the forward is launched the engine can already build the next step’s inputs on the CPU while the GPU is busy, then come back to collect the sampled tokens — overlapping host-side scheduling with device compute.

The annotation itself is built in annotate_profile, which wraps the execute_model call (so the string scopes the forward, not the sampling). The counts come from compute_iteration_details, which buckets each scheduled request — context (prefill: output tokens still 0, including an in-flight chunked-prefill chunk) vs generation (decode):

# vllm/v1/utils.py:470-502 (abbreviated)
for req_id, num_tokens in scheduler_output.num_scheduled_tokens.items():
    if scheduler_output.scheduled_cached_reqs.is_context_phase(req_id) or req_id in new_req_ids:
        num_context_requests += 1;    num_context_tokens += num_tokens      # prefill
    else:
        num_generation_requests += 1; num_generation_tokens += num_tokens   # decode
# -> "execute_context_{ctx_reqs}({ctx_tokens})_generation_{gen_reqs}({gen_tokens})"

What you actually see in the profile. Per scheduler step there are two adjacent regions:

1. execute_context_X(a)_generation_Y(b) — the fused forward. Inside it, per layer: one varlen attention kernel covering both phases, then the big projection/MoE GEMMs with M = a + b (here 9), then the TP all-reduce.
2. sample_tokens — a small tail: the LM-head GEMM on num_reqs rows plus the sampler kernels (softmax / top-k / top-p / multinomial or argmax). It is tiny relative to the dozens of transformer layers, regardless of batch shape.

Read the annotation as the compute profile of that step:

Annotation	What it is	GEMM M	Bound by	All-reduce kernel
context_0(0)_generation_N(N)	pure decode	N (small)	memory bandwidth	trtllm_allreduce_fusion (decode cudagraph)
context_M(big)_generation_0(0)	pure / chunked prefill	large	compute	multimem_all_reduce (eager)
context_1(4)_generation_5(5)	mixed (this example)	4+5=9	mixed	usually the eager path — a prefill token in the batch breaks the pure-decode cudagraph shape

That last row is why a mixed step shows the multimem_all_reduce / larger-M-GEMM signature rather than the decode-only trtllm_allreduce_fusion fused kernel: the decode cudagraph is captured for a uniform query_len == 1 batch, and the moment one request contributes >1 token the runner falls off that captured shape and replays the eager path. (See Part 4 on cudagraph capture for why the captured shapes are decode-only.)

And one practical corollary for the reward-model case: with prefix caching on, context_* counts only the uncomputed tokens, so a 9K-token judge prompt that is a near-total cache hit shows up as context_1(4) — a tiny prefill tail batched alongside whatever decodes are in flight — not the cold context_1(~9000) you’d see on the first, cache-missing call.

Aside — KV cache and prefix caching: the same blocks, deduped

Not a forward-pass stage. Unlike Steps 4–4c, this isn’t a phase that runs in sequence (it doesn’t happen “after sample_tokens”). The KV cache is the data structure the Step 4c forward reads from and writes into every iteration; prefix caching is a dedup layer over it. It’s pulled out here as a standalone aside because the same blocks are touched throughout the forward, not at one point in the timeline.

A common question: is prefix caching a separate cache from the KV cache? No. Prefix caching is not a second store — it is a hash-based dedup-and-reuse layer over the exact same PagedAttention KV blocks. Understanding that one fact explains both how the unified prefill+decode forward reads/writes KV and why the context_1(4) cache-hit forward above is so cheap.

The KV cache is a pool of fixed-size blocks. vLLM v1 carves GPU memory into a BlockPool of KVCacheBlocks — each block holds the K and V for a fixed number of tokens (the block size, e.g. 16). A KVCacheBlock is mostly block_id (its physical slot in the pool), ref_cnt, and an optional _block_hash. Free blocks live in a free_block_queue in eviction (LRU) order. There is one physical pool; “KV cache” and “prefix cache” are two views of it.

How a forward writes KV — slot_mapping. Every request owns a block_table: an ordered list of the physical block_ids holding its tokens’ KV. When the runner prepares a step, it computes a slot_mapping — for each new token in the flattened batch, the exact physical slot (block_id * block_size + offset) where that token’s freshly-computed K/V must be written (compute_slot_mapping). Inside the model, after attention computes K/V for the batch, the reshape_and_cache_flash kernel scatters them into those slots. This is phase-agnostic: a prefill request contributing 4 new tokens writes 4 slots, a decode request contributing 1 token writes 1 slot — the same flattened slot_mapping, the same kernel, in the same forward.

How a forward reads KV — block_table + varlen attention. The attention kernel takes the per-request block_table and seq_lens (alongside the query_start_loc boundaries from Step 4c) and runs in paged, varlen mode: for each request it gathers K/V from that request’s blocks and applies the right mask — a prefill query attends causally over [its already-cached prefix blocks] + [its new tokens]; a decode query (1 token) attends to all of its blocks. One kernel launch serves the whole mixed batch; the block tables are what let prefill and decode coexist, because each request reads only its own blocks regardless of how many new tokens it brought.

Prefix caching is the dedup on those blocks. Two things make it work:

1. Registration. When a request fills a block completely, vLLM hashes the block’s token ids (chained with the previous block’s hash, so the hash encodes the whole prefix up to here) and registers hash -> block in cached_block_hash_to_block (cache_full_blocks).
2. Lookup. A new request’s get_computed_blocks hashes its prompt blocks and calls find_longest_cache_hit. Every hash that matches an existing block means “this prefix is already in the KV pool” — so the request’s block_table is pointed at those existing physical blocks (their ref_cnt is incremented; nothing is copied), and those tokens are marked already computed.

allocate_slots then only allocates new blocks for the uncomputed tail. So a 9K-token judge prompt whose prefix is cached contributes its shared blocks by reference and schedules only the few uncached tokens for compute — that is precisely the context_1(4) forward: 4 new tokens prefilled, while the attention kernel still reads the ~9K cached tokens out of the shared blocks via the block table. (vLLM recomputes the last token even on a full hit — max_cache_hit_length = num_tokens - 1 — because it needs that position’s logits to sample.)

This is why the reward workload (256 jobs × 4 summaries sharing an ~8K job/résumé prefix) is so much cheaper than it looks: the shared prefix is computed once, its full blocks are hash-registered, and the other three summaries of that job reuse the identical physical blocks — ref_cnt bumps, no recompute. Turning prefix caching off doesn’t free a separate cache; it just disables the hash lookup, forcing every summary to recompute and re-store that 8K prefix into fresh blocks (4× the prefill work). Eviction is the same pool’s LRU: when a cached block’s ref_cnt hits zero it goes back on the free_block_queue and can be reclaimed (and its hash entry dropped) when memory is needed.

Step 5 — Result emission

After Sampler returns, the engine builds EngineCoreOutputs (a list of EngineCoreOutput structs, one per Request) containing new_token_ids + new_logprobs (a LogprobsLists of NumPy arrays). msgspec-serialized and pushed to the output socket.

Step 6 — Caller: get_output + OutputProcessor

Back in the caller process, LLMEngine.step calls engine_core.get_output():

# vllm/v1/engine/core_client.py:786-796
def get_output(self):
    outputs = self.outputs_queue.get()  # blocks on Connection.wait + zmq.recv
    ...
    return outputs

This is where most of the wall-clock time goes for short-generation workloads: waiting on IPC. The actual compute happens in the subprocess; the caller just blocks on Connection.wait until it arrives.

Once outputs land, OutputProcessor.process_outputs loops over the N EngineCoreOutput entries and per-Request:

# vllm/v1/engine/output_processor.py:572-687 (abbreviated)
def process_outputs(self, engine_core_outputs, ...):
    request_outputs = []
    for engine_core_output in engine_core_outputs:
        req_state = self.request_states[req_id]

        # 1) IterationStats updates
        self._update_stats_from_output(req_state, engine_core_output, ...)

        # 2) Streamed-text detokenization + stop-string check
        stop_string = req_state.detokenizer.update(new_token_ids, ...)

        # 3) Logprobs accumulation (decodes top-K token IDs to strings!)
        req_state.logprobs_processor.update_from_output(engine_core_output)

        # 4) Build user-facing RequestOutput
        request_output = req_state.make_request_output(...)
        request_outputs.append(request_output)

    return OutputProcessorOutput(request_outputs=request_outputs, ...)

There’s a non-obvious cost in step 3: _update_sample_logprobs calls convert_ids_list_to_tokens which calls tokenizer.decode([tok]) once per logprob entry to populate Logprob.decoded_token:

# vllm/tokenizers/detokenizer_utils.py:83
def convert_ids_list_to_tokens(tokenizer, token_ids):
    token_str_lst = []
    for token_id in token_ids:
        token_str = tokenizer.decode([token_id])
        if token_str is None:
            token_str = ""
        token_str_lst.append(token_str)
    return token_str_lst

If you request many logprobs per position (e.g. logprobs=200 for analysis or constrained-decoding workflows), this dominates OutputProcessor wall time — N requests × K logprobs × M positions HF tokenizer calls per step. For workflows that read only Logprob.logprob numerical values and discard Logprob.decoded_token, this is wasted work.

Part 4 — Architectural Notes Worth Remembering

A few things that bite you if you don’t know them upfront.

n>1 is forbidden with greedy sampling

SamplingParams._verify_greedy_sampling raises ValueError: n must be 1 when using greedy sampling, got N. This is checked in the caller process before the request reaches the engine. The intent is reasonable — n=k greedy gives k identical sequences — but if you have a custom LogitsProcessor that produces row-distinct outputs (e.g. for beam search initialization), you can work around with a tiny temperature:

# temperature must be >= _SAMPLING_EPS (1e-5).
# vLLM clamps any temperature < 0.01 up to 0.01 with a warning.
SamplingParams(n=200, temperature=1e-3, max_tokens=K)

After temperature scaling, sharp logit distributions collapse to deterministic argmax anyway.

logprobs_mode matters for any LogitsProcessor that masks

Default logprobs_mode="raw_logprobs" reports logprobs computed BEFORE your LogitsProcessor runs. If your LP masks the logit space (sets invalid tokens to -inf), the reported logprobs include the masked-out tokens. Use logprobs_mode="processed_logprobs" to report post-mask values, which is almost always what you want for constrained decoding.

Spawn hooks: sitecustomize.py is the canonical place

If you need to install a behavior change (a monkey-patch, a counter, a metric exporter) that needs to run inside the spawned EngineCore subprocess, the canonical mechanism is a sitecustomize.py on PYTHONPATH. Python’s site module loads it at every interpreter startup, before the user’s imports. Combined with an env-var gate (which the child inherits from the parent), you get clean opt-in behavior:

# /path/to/my_patches/sitecustomize.py
import os
if os.environ.get("MY_PATCH") == "1":
    # ... install patches before vllm imports
    pass

PYTHONPATH=/path/to/my_patches:$PYTHONPATH MY_PATCH=1 python my_script.py

This also works for vLLM’s own EngineCore — the env var is propagated to the spawned child by mp.Process, and sitecustomize.py runs before vLLM’s imports inside the child.

IPC dominates short-generation latency

For workflows that issue many llm.generate(...) calls with short generations (a few decode steps each), you’re going to spend most of your wall-clock time inside Connection.wait + zmq.poll waiting for the engine output. The two-process design pays off for online serving (where the engine continuously batches across many concurrent requests), but for offline short-burst workloads the per-call IPC tax is real.

If your workload genuinely needs many small llm.generate calls, three options exist:

1. Batch more per call. Combining 10 sequential calls of 20 prompts each into one call of 200 prompts amortizes the IPC over more work.
2. Use AsyncLLM if you have natural concurrency (multiple producers).
3. Run EngineCore in-process via InprocClient. This is documented as “for debugging” in the source but is functionally usable for offline workloads. You lose the GIL isolation but eliminate the IPC.

LLM does not auto-clean

When your LLM object goes out of scope, its __del__ triggers MPClient.shutdown which terminates the EngineCore subprocess. If your script crashes mid-run, the subprocess can be left orphaned (Linux init will eventually reap it, but you may see a stale process around for a while). For long-running scripts, explicit shutdown via del llm or context manager is good hygiene.

Part 5 — The Online Server Path (vllm serve)

Everything above traced the offline LLM(...) path. The OpenAI-compatible server (vllm serve) reuses the exact same EngineCore — same subprocess, same model load, same CUDA-graph capture. What changes is only the top: a uvicorn HTTP app + per-endpoint serving handlers (the frontend), an AsyncMPClient (asyncio + ZMQ, instead of the blocking SyncMPClient), and request fan-in — each HTTP request becomes one async generator multiplexed over the shared engine.

The permalinks in this part are pinned to v0.22.0 (the offline sections above use an earlier commit); the line numbers are from a 0.22.0 install.

Phase A — Startup: vllm serve → a live engine

# vllm/entrypoints/cli/serve.py:148
uvloop.run(run_server(args))

run_server opens the listen socket, then run_server_worker enters build_async_engine_client, which builds the engine:

# vllm/entrypoints/openai/api_server.py:136
async_llm = AsyncLLM.from_vllm_config(vllm_config=vllm_config, ...)

AsyncLLM.__init__ is where online and offline converge. Its one structural line:

# vllm/v1/engine/async_llm.py:146
self.engine_core = EngineCoreClient.make_async_mp_client(...)

make_async_mp_client returns an AsyncMPClient rather than the offline SyncMPClient — but both go through the same launch_core_engines spawn path from Part 2. So the EngineCore subprocess boots identically: load weights, profile KV cache, capture CUDA graphs, ready-handshake. By the time AsyncLLM exists, the GPU is fully warmed and graph-captured.

AsyncLLM then starts a background output handler coroutine — the pump that drains engine outputs into per-request queues:

# vllm/v1/engine/async_llm.py:170
self.output_handler: asyncio.Task | None = None
...  self._run_output_handler()   # async_llm.py:373

Finally init_app_state wires the engine into the FastAPI serving handlers and uvicorn starts accepting requests. /health only flips to 200 after all of the above — which is why server startup is dominated by model load + graph capture.

Phase B — Handling a request

A POST /v1/completions lands here:

# vllm/entrypoints/openai/completion/api_router.py:46
async def create_completion(request, raw_request):
    handler = completion(raw_request)
    generator = await handler.create_completion(request, raw_request)
    ...
    return StreamingResponse(content=generator, media_type="text/event-stream")

The handler renders the prompt and, per prompt, calls into the engine:

# vllm/entrypoints/openai/completion/serving.py:204
generator = self.engine_client.generate(engine_input, sampling_params, request_id_item, ...)

AsyncLLM.generate is an async generator. Under the hood it:

1. creates a per-request asyncio.Queue and registers it with the output processor (_add_request),
2. pushes the request to the engine over ZMQ —

   # vllm/v1/engine/async_llm.py:412
   await self.engine_core.add_request_async(request)
   

3. yields RequestOutputs as the background output handler demuxes EngineCoreOutputs into that request’s queue.

Unlike offline — where LLM.generate submits a batch and blocks in zmq.poll until it finishes — the online path submits one request at a time, and the engine’s scheduler (Part 3, Step 3) continuously batches whatever requests are in flight across all concurrent connections. That continuous batching is the entire reason for the two-process design; the per-call IPC tax that hurts offline short-bursts is amortized away here.

Phase C — Where/when CUDA graphs are captured (same place, once)

CUDA graphs are not captured per request and not on the server frontend. They’re captured once, at engine startup, inside the EngineCore subprocess — the same Phase-5b step the offline path hits:

EngineCore.__init__
  └─ self.model_executor.initialize_from_config(...)   # vllm/v1/engine/core.py:286
       └─ Worker.compile_or_warm_up_model()            # vllm/v1/worker/gpu_worker.py:572
            └─ GPUModelRunner.capture_model()          # vllm/v1/worker/gpu_model_runner.py:6373
                 with graph_capture(device=...):       # gpu_model_runner.py:6416
                     # replay dummy runs at each cudagraph_capture_size,
                     # recording one CUDA graph per batch-shape bucket

In the server log it appears as Capturing CUDA graphs (...) followed by Graph capturing finished in N secs. It runs before /health returns 200 (and enforce_eager=True skips the whole step — trading startup time for ~no graph at run time). At request time the engine just replays the captured graph that matches the batch’s shape — there is no capture on the hot path.

This is exactly what a server-side profile sees. Hitting the running server’s POST /start_profile (enabled by --profiler-config.profiler=torch --profiler-config.torch_profiler_dir=...) and driving a few requests captures the graph replays across every TP/EP rank — the prefill/decode forwards, the all-reduces, the MoE GEMMs — never the one-time capture. The offline LLM profiler and the online /start_profile land on the same worker kernels, because they share the same EngineCore; only the frontend (a separate async_llm trace on the online side) differs.

Summary: A Mental Model

A useful mental model for vLLM v1:

• Two processes, one ZMQ pipe. Caller-side: tokenize, render, build RequestState, push EngineCoreRequest, wait for output, build RequestOutput. Engine-side: schedule, forward, sample, emit EngineCoreOutput.
• Six layers, with a clean boundary at layer 3/4. Everything above the boundary is Python orchestration; everything below is GPU-driving + scheduler bookkeeping.
• One Request = one row. Parallel sampling fans out to N independent Requests at the LLMEngine layer; the engine has no concept of “one Request, N rows”.
• Init is one-time. ~14 s for a typical model — model load, KV profiling, CUDA graph capture, ready handshake. Then the subprocess stays alive across all generate calls.
• Spawn semantics gate every patch. Anything you want to take effect in EngineCore needs to be installed via sitecustomize.py + env-var gate, not runtime monkey-patch in the parent.

If you remember these five points, the rest of vLLM v1’s behavior makes sense when you encounter it. The codebase is large but the architecture is clean.

Happy hacking.