SGLang Deep Dive: How Engine.generate() Boots and Runs

Introduction

SGLang is a high-performance LLM serving stack built around RadixAttention and a custom scheduler. The public Python API is small:

from sglang import Engine

engine = Engine(model_path="meta-llama/Llama-3.1-8B-Instruct")
out = engine.generate(
    "What is the capital of France?",
    sampling_params={"max_new_tokens": 64, "temperature": 0.7},
)
print(out["text"])

…but the internals are a different beast. SGLang ships a three-process architecture: a tokenizer-manager process (where your script lives), a scheduler process (which owns the model + KV cache + RadixAttention tree), and a detokenizer process (which converts output token IDs back to text). All three communicate over ZeroMQ.

This post traces what actually happens when you call Engine(...) and then engine.generate(...), with permalinks pinned to commit 0f21fe9 on main.

Focus is the offline Engine path — the same engine internals also serve the OpenAI-compatible HTTP server and the SGL frontend DSL, but the offline path is the cleanest entry point to read the code.

🧭 Companion piece This post mirrors the structure of an earlier vLLM v1 deep-dive. vLLM v1 uses a 2-process architecture; SGLang uses 3. The differences in design are illustrative — same underlying problem (run a model behind an IPC boundary so the model loop isn’t blocked by the framework loop), different splits.

Part 1 — The Three-Process Architecture

The whole engine looks like this:

              user process                                 scheduler subprocess                         detokenizer subprocess
  ┌──────────────────────────────────┐               ┌────────────────────────────────┐         ┌────────────────────────────┐
  │ Engine.generate(prompt, params)  │               │ event_loop_overlap (busy loop) │         │ DetokenizerManager         │
  │  ↓                               │               │  - recv_requests via ZMQ       │         │  event_loop:               │
  │ TokenizerManager                 │               │  - schedule next batch         │         │   - recv_obj from ZMQ      │
  │  - tokenize prompt               │     ZMQ       │  - run model forward           │   ZMQ   │   - tokenizer.batch_decode │
  │  - assign rid                    │ ──── send ──▶ │  - sampler + custom LP         │ ──────▶ │   - send_to_tokenizer_     │
  │  - send GenerateReqInput         │               │  - emit BatchTokenIDOutput     │         │     manager                │
  │                                  │               │  - update RadixAttention tree  │         └────────────────────────────┘
  │  await response_event            │               │                                │                       │
  │                                  │      ZMQ      │  Scheduler.event_loop_overlap  │                       │
  │  resolve rid → result            │ ◀──── recv ── │   recv ←────────────────────── │                       │
  └──────────────────────────────────┘               └────────────────────────────────┘                       │
        ▲                                                                                                       │
        └──────────────── ZMQ (BatchStrOutput with text) ───────────────────────────────────────────────────────┘

Three things are worth noting before we walk through the code:

Why a separate detokenizer process at all? vLLM v1 collapses detokenization into the caller process (OutputProcessor lives in the user process). SGLang fans it out. The reason is throughput at scale: HF tokenizer’s batch_decode is CPU-bound and surprisingly expensive at high beam counts (we’ll come back to this). Moving it to its own process means the scheduler’s busy loop never waits on tokenizer work.

Why ZMQ? The same reason vLLM uses it: you need IPC that doesn’t go through the GIL, and ZMQ’s send_multipart + recv_multipart give you message-framing for free. SGLang uses three ZMQ channels:

• tokenizer_ipc_name: user → scheduler (incoming requests)
• scheduler_input_ipc_name: scheduler → detokenizer (token-ID outputs)
• detokenizer_ipc_name: detokenizer → tokenizer-manager (text outputs)

Why “spawn” not “fork”? CUDA-initialized processes can’t be forked safely, and SGLang sets mp.set_start_method("spawn", force=True) at engine init time (engine.py:1192). The implication for hooking is the same as vLLM: runtime monkey-patches in your script don’t propagate to the spawned children. The canonical hook is a sitecustomize.py on PYTHONPATH.

Core layer permalinks

Component	Class	File:line
User-facing entry	Engine	python/sglang/srt/entrypoints/engine.py:144
User-process tokenizer / IPC client	TokenizerManager	python/sglang/srt/managers/tokenizer_manager.py:215
Scheduler-subprocess body	Scheduler	python/sglang/srt/managers/scheduler.py:317
Scheduler busy loop	event_loop_overlap	scheduler.py:1412
Scheduler entry function (subprocess target)	run_scheduler_process	scheduler.py:3758
Detokenizer-subprocess body	DetokenizerManager	detokenizer_manager.py:73
Sampler + LP dispatch	Sampler.forward	sampler.py:77

Part 2 — Initialization: From Engine(...) to “Ready”

What actually happens when you write engine = Engine(model_path="...")?

Wall time on a typical setup (one H100 80GB, 1.7B-parameter Qwen3-class model, default config): about 10–15 seconds end-to-end, dominated by model weight loading and CUDA graph capture in the spawned scheduler subprocess.

Let’s walk it phase by phase.

Phase 1 — User-process Engine construction

Engine.__init__ at engine.py:165 does the parent-process work first:

# python/sglang/srt/entrypoints/engine.py:170-205 (abbreviated)
def __init__(self, **kwargs):
    load_plugins()                                          # 1) plugin entrypoints

    server_args = self.server_args_class(**kwargs)          # 2) parse kwargs into ServerArgs
    self.server_args = server_args

    self.tokenizer_manager = None                           # 3) pre-init for atexit safety
    atexit.register(self.shutdown)                          # 4) clean shutdown hook

    # 5) THE BIG STEP — spawn child subprocesses
    (
        tokenizer_manager,
        template_manager,
        port_args,
        scheduler_init_result,
        subprocess_watchdog,
    ) = self._launch_subprocesses(...)

_launch_subprocesses is where the actual fork-and-spawn happens. Three children are about to come into existence:

1. N scheduler processes (one per GPU rank if TP > 1)
2. 1 detokenizer process
3. The tokenizer-manager stays in the parent process (it’s a Python object, not a subprocess)

Phase 2 — mp.set_start_method("spawn") and why it matters

Before any subprocess is launched, SGLang forces the spawn method (engine.py:1192):

# python/sglang/srt/entrypoints/engine.py:1192
mp.set_start_method("spawn", force=True)

This is the canonical “force spawn so CUDA-initialized processes can fork safely” pattern. It’s also what gates the hookability story: spawned children re-import everything from clean Python state. If you’ve monkey-patched something in your top-level script, that patch does NOT propagate.

🔥 Hook point: sitecustomize.py Spawned children DO inherit the parent’s environment variables, and Python’s site module loads sitecustomize.py at every interpreter startup — before any sglang import. This is the canonical place to install patches that need to take effect inside the scheduler / detokenizer subprocesses.

Pattern:

PYTHONPATH=/path/to/my_patches:$PYTHONPATH MY_PATCH_FLAG=1 python my_script.py

with my_patches/sitecustomize.py reading MY_PATCH_FLAG and gating its installs.

Phase 3 — The Spawn Sites

This is the bit worth knowing precisely. SGLang spawns subprocesses in two places.

Spawn site #1 — scheduler processes (one per TP rank). At engine.py:571-577:

# python/sglang/srt/entrypoints/engine.py — inside _launch_scheduler_processes
proc = mp.Process(
    target=run_scheduler_process_func,                    # ← target fn
    args=(server_args, port_args, gpu_id, tp_rank,
          attn_cp_rank, moe_dp_rank, moe_ep_rank, pp_rank,
          None, writer),                                  # `writer` is the readiness pipe
)
with memory_saver_adapter.configure_subprocess(),
     numa_utils.configure_subprocess(server_args, gpu_id):
    proc.start()

run_scheduler_process_func is the static function run_scheduler_process in scheduler.py. We’ll trace what it does in the next subsection.

Spawn site #2 — detokenizer process (single, regardless of TP). At engine.py:728-736:

# python/sglang/srt/entrypoints/engine.py — inside _launch_subprocesses
detoken_proc = mp.Process(
    target=run_detokenizer_process_func,                  # ← run_detokenizer_process
    args=(server_args, port_args),
)
detoken_proc.start()

That’s it — that’s where the multi-process architecture actually comes into being. Two mp.Process(...) calls with mp.set_start_method("spawn") already set above them.

Phase 4 — In the Spawned Scheduler Subprocess

The scheduler subprocess’s entry point is run_scheduler_process:

# python/sglang/srt/managers/scheduler.py — abbreviated
def run_scheduler_process(server_args, port_args, gpu_id, tp_rank, ..., pipe_writer):
    # 1) Set process title for ps / top visibility
    set_process_title(...)

    # 2) Build the actual Scheduler object — this loads the model
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, ...)

    # 3) Signal readiness to the parent via the pipe
    pipe_writer.send({"status": "ready", ...})
    pipe_writer.close()

    # 4) Drop into the busy loop — never returns until shutdown
    if scheduler.enable_overlap:
        scheduler.event_loop_overlap()
    else:
        scheduler.event_loop_normal()

The Scheduler.__init__ constructor is where the heavy lifting happens:

1. Load the model weights (~few seconds — typically the dominant init cost)
2. Initialize the KV cache pool + RadixAttention tree
3. Capture CUDA graphs (the Capturing CUDA graphs ... 100% [N/N] log lines) — comparable timing to vLLM
4. Set up the three ZMQ sockets (recv from tokenizer-manager, send to detokenizer, internal scheduler)
5. Initialize structured-output backends if grammar usage is configured (xgrammar / outlines / llguidance)

Once pipe_writer.send({"status": "ready"}) fires, the parent’s scheduler_init_result.wait_for_ready() returns and the parent knows this scheduler is up.

Phase 5 — In the Spawned Detokenizer Subprocess

The detokenizer is much simpler — just a tokenizer + ZMQ receive loop. From detokenizer_manager.py:73:

# python/sglang/srt/managers/detokenizer_manager.py
class DetokenizerManager(MultiHttpWorkerDetokenizerMixin):
    def __init__(self, server_args, port_args):
        self.tokenizer = get_tokenizer(...)
        # Set up ZMQ pull from scheduler, push to tokenizer-manager
        ...

    def event_loop(self):
        while True:
            recv_obj = self.recv_from_scheduler.recv_pyobj()
            output = self._decode_batch_token_id_output(recv_obj)
            self.send_to_tokenizer_manager.send_pyobj(output)

The _decode_batch_token_id_output method is the hot loop — detokenizer_manager.py:217. It converts a batch of token IDs into text via HF tokenizer’s batch_decode. We’ll see in Part 3 why this is sometimes the surprising bottleneck.

Phase 6 — Tokenizer-Manager Construction (Stays in Parent)

The tokenizer-manager runs in the parent process. After both subprocesses signal ready, the parent constructs it via init_tokenizer_manager_func:

# python/sglang/srt/entrypoints/engine.py
if server_args.tokenizer_worker_num == 1:
    tokenizer_manager, template_manager = init_tokenizer_manager_func(
        server_args, port_args
    )
else:
    tokenizer_manager = MultiTokenizerRouter(server_args, port_args)
    template_manager = None

TokenizerManager.__init__ sets up:

• The HF tokenizer (loaded from disk into the parent process — yes, two copies of the tokenizer end up in memory: one in the parent’s TokenizerManager, one in the detokenizer subprocess).
• ZMQ sockets: send to scheduler, recv from detokenizer.
• An async event loop and a per-rid (request-ID) → asyncio.Future map.
• A bunch of sampling-param constructors.

Phase 7 — Final Wiring + Return

At this point all three processes are alive, all six ZMQ sockets are bound, and the tokenizer is loaded. Control returns to Engine.__init__, which does some last book-keeping and then returns. The user’s script can call engine.generate(...).

Part 3 — Runtime: One engine.generate(...) Call

After init, engine.generate(...) is the hot path. Trace:

user process                                    scheduler subprocess                    detokenizer subprocess
─────────────                                   ──────────────────                      ──────────────────────
Engine.generate(prompt, sampling_params)
 ↓
TokenizerManager.generate_request                    busy loop (event_loop_overlap)              busy loop (event_loop)
 ├─ tokenize(prompt)                                  ├─ recv_requests via ZMQ                    ├─ recv from scheduler
 ├─ assign rid                                        ├─ process_input_requests                   ├─ tokenizer.batch_decode
 ├─ build GenerateReqInput                            ├─ get_next_batch_to_run                    └─ send to tokenizer_mgr
 └─ send via ZMQ ──────────────────────────────▶     ├─ run_batch (model forward)                            │
                                                      ├─ Sampler.forward                                       │
                                                      │   └─ apply_custom_logit_processor(logits)              │
                                                      ├─ process_batch_result                                  │
                                                      └─ send BatchTokenIDOutput ───────▶                      │
 await rid → result                                                                                            │
                                                                                                                ▼
 resolve future ◀──────── ZMQ (BatchStrOutput with decoded text) ◀──────────────────────────────────────────

Step 1 — Tokenize + Send

The user’s Engine.generate(prompts, ...) call resolves through TokenizerManager.generate_request at line 507:

# python/sglang/srt/managers/tokenizer_manager.py — abbreviated
async def generate_request(self, obj, request=None):
    rid = obj.rid or str(uuid.uuid4())                    # 1) assign request ID

    # 2) tokenize the prompt(s) inline (synchronous CPU work)
    input_ids = self.tokenizer.encode(obj.text)

    # 3) build a GenerateReqInput msg (or similar wire type) and send to scheduler
    self.send_to_scheduler.send_pyobj(req_input)

    # 4) register a future, await it
    state = ReqState(...)
    self.rid_to_state[rid] = state
    return await state.future

For a batch generate (e.g., beam search at the user side), _handle_batch_request at tokenizer_manager.py:1363 loops and submits one ZMQ message per prompt.

🧠 Subtle: SGLang sends one ZMQ payload per prompt via separate send_pyobj calls — there’s no “batched submit” message in the wire protocol. The scheduler accumulates them on its recv_requests side. This is the same per-Request orchestration overhead pattern vLLM has, with a fundamental similarity: the engine treats each beam / candidate as an independent request.

Step 2 — Scheduler Receives + Schedules

Inside the scheduler subprocess, the busy loop is Scheduler.event_loop_overlap:

# python/sglang/srt/managers/scheduler.py:1412
def event_loop_overlap(self):
    """A scheduler loop that overlaps the CPU processing and GPU computation."""
    self.result_queue: Deque[...] = deque()

    def pop_and_process():
        tmp_batch, tmp_result = self.result_queue.popleft()
        self.process_batch_result(tmp_batch, tmp_result)

    while True:
        recv_reqs = self.recv_requests()                  # 1) drain input ZMQ socket
        self.process_input_requests(recv_reqs)            #    add new reqs to waiting queue
        if self._engine_paused:
            continue

        batch = self.get_next_batch_to_run()              # 2) pick next batch (running + waiting)
        ...
        self.run_batch(batch)                             # 3) execute model forward + sampler
        ...
        if disable_overlap_for_batch:
            pop_and_process()                             # 4) post-process the previous batch

get_next_batch_to_run is where SGLang’s RadixAttention shines: it consults the prefix-tree to figure out which prefixes are already cached, and groups requests with shared prefixes for free (zero KV recomputation). The scheduler is the single most complex file in sglang — about 4,000 lines.

Step 3 — Forward Pass + Sampling

run_batch calls into the model runner, which executes the forward pass on the assembled batch (using PagedAttention / RadixAttention for KV) and produces a (num_rows, vocab) logits tensor.

The logits then flow into Sampler.forward:

# python/sglang/srt/layers/sampler.py — abbreviated
def forward(self, logits_output, sampling_info, ...):
    logits = logits_output.next_token_logits

    # 1) preprocess: custom LP, NaN guard, grammar bitmask
    logits = self._preprocess_logits(logits, sampling_info)

    # 2) apply temperature, top-k, top-p
    ...

    # 3) sample tokens
    sampled_token_ids = ...

_preprocess_logits (sampler.py:58) is the constraint-decoding path:

# python/sglang/srt/layers/sampler.py:58
def _preprocess_logits(self, logits, sampling_info):
    if sampling_info.has_custom_logit_processor:
        apply_custom_logit_processor(logits, sampling_info)        # ← user-supplied LP
    if sampling_info.has_grammar:
        ...                                                         # ← xgrammar / outlines bitmask
    return logits

This is where a user-registered CustomLogitProcessor runs. Each request can carry its own per-row state (via sampling_params.custom_params), and the LP is invoked on the full batched logits tensor. Note: apply_custom_logit_processor is itself defined at sampler.py:712.

Step 4 — Result Accumulation + Send to Detokenizer

After sampling, the scheduler emits a BatchTokenIDOutput (token IDs + metadata, no decoded text yet) and pushes it to the detokenizer process via ZMQ. The user-process tokenizer-manager has already attached to the detokenizer’s output socket and is waiting.

Step 5 — Detokenization (in its Own Process)

The detokenizer’s event_loop (detokenizer_manager.py:137) wakes up on each incoming batch:

# python/sglang/srt/managers/detokenizer_manager.py:137
def event_loop(self):
    while True:
        recv_obj = self.recv_from_scheduler.recv_pyobj()        # 1) get token IDs
        output = self._decode_batch_token_id_output(recv_obj)   # 2) tokenizer.batch_decode
        self.send_to_tokenizer_manager.send_pyobj(output)       # 3) push text to user

For high-beam-count workloads (e.g., beam=200, max_tokens=3 → ~600 detok calls per beam-search invocation), _decode_batch_token_id_output becomes a measurable cost — comparable in magnitude to vLLM’s convert_ids_list_to_tokens issue. The fix when you don’t need decoded text is to short-circuit this method via a sitecustomize patch.

Step 6 — User Process Receives Result

The tokenizer-manager has an async receive loop that resolves rid-keyed futures. When the detokenizer’s text output lands, the user’s await state.future returns and engine.generate(...) returns the result dict.

Part 4 — Architectural Notes Worth Remembering

Things that bite you if you don’t know them upfront.

engine.generate is per-Request, no native beam search

Up to and including commit 0f21fe9 (sglang main as of this writing), there is no native Engine.beam_search() API. To do beam search:

• Submit N prompts (one per beam state) per step via batched generate
• Read each request’s logprobs back, pick top-K across beams in user-side Python
• Submit the next step’s N prompts

This is the same pattern needed in vLLM for STATIC-style constraint decoding, because vLLM’s LLM.beam_search doesn’t pass logits_processors (filed as a known gap). PR #15645 is in flight to add native beam search to sglang via a Mixin pattern, but as of writing it’s open and explicitly disables grammar with beam search.

Three-way IPC means three places to look for hot Python loops

When debugging a slow sglang call:

symptom	likely culprit
Big share of wall in mp.Connection.wait / zmq.poll in the user process	Scheduler is busy running model forward; expected for short generations
Big share of wall in _decode_batch_token_id_output in the detokenizer	High beam count + you don’t read text → patch via sitecustomize
Big share of wall in process_batch_result in the scheduler	Per-Request bookkeeping (logprob accumulation, output_token append)
Big share of wall in apply_custom_logit_processor in the scheduler	Your LP is slow; profile inside the LP itself

Custom LogitsProcessor runs in the scheduler subprocess

Because the scheduler is spawned, your CustomLogitProcessor class is re-imported in the child. State you set on it in the parent does NOT survive. Pass per-request state via sampling_params.custom_params — these are picklable and shipped with each request over ZMQ.

sitecustomize.py propagates through mp.spawn()

Same trick that works in vLLM works here. Spawned children inherit the parent’s environment variables, and site.py loads sitecustomize.py before any vLLM/sglang imports. Combined with an env-var gate, you get clean opt-in patches that propagate to all three sglang processes.

This is how I shipped a “skip detok work the caller never reads” patch for a constraint-decoding workload that didn’t need decoded text — measured ~10–16% wall savings at mid beam widths. Pattern:

# /path/to/my_patches/sitecustomize.py
import os
if os.environ.get("MY_SGLANG_PATCH") == "1":
    # Patch DetokenizerManager._decode_batch_token_id_output to no-op
    from sglang.srt.managers.detokenizer_manager import DetokenizerManager
    def _patched(self, recv_obj):
        return [""] * len(recv_obj.rids)
    DetokenizerManager._decode_batch_token_id_output = _patched

Run with PYTHONPATH=/path/to/my_patches:$PYTHONPATH MY_SGLANG_PATCH=1 python my_script.py. The patch installs in the user process, the scheduler subprocess (re-imports), and the detokenizer subprocess (re-imports), all gated on the same env var.

RadixAttention is what makes sglang fast for repeated prefixes

The scheduler maintains a prefix tree of cached KV blocks and looks up incoming requests’ prompts against it. Two requests with the same chat-history prefix share KV pages. This is sglang’s signature optimization and is why it dominates benchmarks where the workload has repeated structure (chat with a long system prompt, JSON tool calling, beam search across a shared prompt — anything where many requests share leading tokens).

The implementation is in python/sglang/srt/mem_cache/radix_cache.py — worth reading if you’re optimizing serving workloads. It’s the design choice that most distinguishes sglang from vLLM (which uses block-hash deduplication for the same purpose, but doesn’t maintain an explicit tree for prefix lookup).

Summary: A Mental Model

A useful mental model for SGLang:

• Three processes, one ZMQ mesh. User-process tokenizes and dispatches; scheduler subprocess owns model + KV + RadixAttention; detokenizer subprocess converts token IDs back to text. Six ZMQ sockets total.
• Scheduler is the brain. RadixAttention prefix-cache lookup, batch packing, model forward, sampler, output emission — all in one subprocess.
• Detokenizer is for throughput, not correctness. It exists because moving tokenizer.batch_decode out of the scheduler’s hot loop matters when beam counts / batch sizes get big.
• Engine.generate is per-Request. Beam search is your problem. Submit N prompts per step, pick top-K in Python. Or wait for #15645 to land.
• Spawn semantics gate every patch. sitecustomize.py + env-var gate is the canonical hook. Same pattern as vLLM.

If you remember these five points, the rest of sglang’s behavior makes sense when you encounter it. The codebase is large but the architecture is clean.

---

🔗 If you read the vLLM v1 deep-dive first, you may find the contrast useful: vLLM v1 = 2 processes, sglang = 3 processes. Both spawn via mp.set_start_method("spawn"), both use ZMQ, both have the same “pre-tokenize in parent, model loop in child” split. SGLang’s extra detokenizer process is the visible architectural difference; RadixAttention is the invisible one.

Happy hacking.