GRPO from First Principles, and How verl Implements It

All verl code permalinks in this post are pinned to commit 9f73954a of verl-project/verl so the line numbers stay valid. main will drift; substitute main for 9f73954a in any link to get the latest version.

Why GRPO

Pre-2024, the standard recipe for training LLMs with RL was PPO with a learned value function. RLHF papers from OpenAI, Anthropic, Meta — all PPO. Each training step requires four model passes:

1. Actor generates rollouts (the policy being trained).
2. Reference policy computes log-probs of those rollouts (frozen SFT, used for KL).
3. Reward model scores the rollouts.
4. Critic predicts the expected return for the same rollouts.

The critic is a separate transformer the same size as the actor. Training it (its own gradients, optimizer state, GPUs) typically doubles the cost of an RL run versus pure supervised fine-tuning. And the critic is famously hard to stabilize — wrong-signed advantages early in training tank the actor.

In February 2024, the DeepSeekMath paper (Shao et al.) showed you can drop the critic entirely if you sample multiple rollouts per prompt and use their mean as a baseline. They named the variant GRPO — Group Relative Policy Optimization. It produced state-of-the-art math reasoning at a fraction of PPO’s compute. A year later, DeepSeek-R1 reproduced the result on general reasoning, and GRPO became the de facto default for open-weight RL training.

This post derives GRPO from PPO, shows where the math lives in verl (one of the most-used open-source RL frameworks), and walks through the architecture that lets a single Ray job run actor training + reward scoring + rollout generation across an 8+ GPU cluster.

Recap: PPO in one screen

PPO (Schulman et al. 2017) optimizes a clipped surrogate of the policy gradient. Let $\pi_\theta$ be the current policy, $\pi_{\theta_{\text{old}}}$ the policy that generated the rollouts in this batch, $A_t$ the advantage at token $t$, and $r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$ the importance ratio. The PPO clipped objective is:

\[\mathcal{L}^{\text{PPO}}(\theta) \;=\; \mathbb{E}_t\!\left[\,\min\!\Big(r_t(\theta)\,A_t,\;\;\text{clip}\!\big(r_t(\theta),\,1-\varepsilon,\,1+\varepsilon\big)\,A_t\Big)\right]\]

Three pieces matter for the GRPO derivation:

1. The ratio $r_t$ keeps the update off-policy-safe — we can take multiple gradient passes over a single batch of rollouts because $r_t$ corrects for the distribution shift between $\pi_{\theta_{\text{old}}}$ and the current $\pi_\theta$.
2. The clip bounds the per-token update magnitude, preventing the kind of large step that collapses the policy.
3. The advantage $A_t$ is computed from the reward signal and a baseline. In PPO this is done with generalized advantage estimation (GAE), which requires a critic $V_\phi(s_t)$ that predicts expected return.

It is (3) that GRPO replaces.

GRPO: the group-mean baseline

The insight is simple and almost embarrassing in retrospect. If for each prompt $q$ we sample $n$ rollouts ${o_1, \dots, o_n} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)$ and obtain scalar rewards $R_1, \dots, R_n$, then the group mean $\bar{R} = \frac{1}{n}\sum_i R_i$ is already a perfectly good baseline for the policy gradient — and it is unbiased, since it depends only on the prompt $q$.

GRPO defines the advantage of rollout $i$ as:

\[A_i \;=\; \frac{R_i - \bar{R}}{\sigma_R + \epsilon} \qquad \text{where } \bar{R} = \tfrac{1}{n}\sum_j R_j, \;\; \sigma_R = \text{std}(R_1, \dots, R_n).\]

The standard-deviation normalization (controlled by algorithm.norm_adv_by_std_in_grpo in verl) is a variance-reduction trick — without it, prompts with high reward variance dominate the gradient. Some implementations keep the std off and just center; in verl the choice is config-driven.

Every token in rollout $i$ receives the same advantage $A_i$. So the GRPO surrogate is just PPO with this token-broadcast advantage substituted in:

\[\mathcal{L}^{\text{GRPO}}(\theta) \;=\; \mathbb{E}_q \!\left[\frac{1}{n}\sum_{i=1}^n \frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\min\!\Big(r_{i,t}(\theta)\,A_i,\,\text{clip}(r_{i,t}(\theta),\,1-\varepsilon,\,1+\varepsilon)\,A_i\Big)\right]\]

That’s the entire algorithm. No critic. No value loss. No GAE bootstrapping. The mean over $n$ rollouts gives you a baseline; everything else inherits from PPO.

In verl this advantage is computed in verl/trainer/ppo/core_algos.py:267–331 by compute_grpo_outcome_advantage(). The function takes per-rollout token-level rewards (typically just an end-of-sequence scalar), groups them by prompt index, centers (and optionally normalizes), and broadcasts the result back to token level so it can plug into the existing PPO loss path. The exact lines are walked through in §”Where the math lives in verl: a code tour” below.

Why dropping the critic works

The policy gradient theorem says any baseline $b(q)$ that depends only on the prompt (not on the action) leaves the gradient unbiased:

\[\nabla_\theta\, \mathbb{E}_{o \sim \pi_\theta}[R(o)\,|\,q] \;=\; \mathbb{E}_o[(R(o) - b(q))\,\nabla_\theta \log \pi_\theta(o\,|\,q)]\]

PPO’s learned $V_\phi(s_t)$ is one choice of baseline; the group mean $\bar{R}_q$ is another. Both unbiased. The question is variance.

A perfect critic gives a per-state baseline that maximally reduces variance. A group mean gives a per-prompt baseline that’s less granular — but the cost of computing it is one extra rollout per prompt, vs. an entire transformer-sized critic. As long as $n$ is large enough that $\bar{R}_q$ is a reasonable estimate of $\mathbb{E}_o[R(o)\,

\,q]$, GRPO wins on compute-per-effective-gradient.

What’s “large enough”? The DeepSeekMath paper used $n=64$ on math problems. Most LLM-RL setups today use $n=4$ to $n=8$. With $n=2$, the centered reward is always $\pm

R_1 - R_2

/2$, which is informative but very noisy. With $n=1$ the advantage is identically zero and the policy gradient is dead — so $n \geq 2$ is hard-required.

There’s a related question: GRPO’s “advantage” is constant across the whole sequence (token-broadcast from the rollout-level $A_i$). Doesn’t that lose information about which tokens contributed to the reward? Yes — and that’s the trade. For tasks with a sparse end-of-sequence reward (math answer correct/not, judge says yes/no), there’s no fine-grained per-token credit to assign anyway, so the broadcast costs nothing. For tasks with dense rewards (e.g., per-token preference signals), GRPO is genuinely a worse fit.

The KL term: K3, not naive log-ratio

RL on LLMs almost always anchors the actor to a frozen reference policy $\pi_{\text{ref}}$ (usually the SFT checkpoint). Without that, the actor drifts arbitrarily — RL is happy to mode-collapse onto whatever the reward signal rewards, even gibberish that exploits the reward model.

The standard recipe adds $\beta \cdot \text{KL}(\pi_\theta | \pi_{\text{ref}})$ to the loss. The naive Monte Carlo estimate of KL from a single sample is:

\[\widehat{\text{KL}}_{\text{naive}} = \log \pi_\theta(o) - \log \pi_{\text{ref}}(o)\]

This is unbiased but high-variance — it can even go negative on individual samples. John Schulman’s KL approximation note introduced a low-variance unbiased alternative (the “K3” estimator):

\[\widehat{\text{KL}}_{\text{K3}}(\theta) \;=\; \mathbb{E}_o\!\left[\exp\!\big(\log \pi_{\text{ref}}(o) - \log \pi_\theta(o)\big) - \big(\log \pi_{\text{ref}}(o) - \log \pi_\theta(o)\big) - 1\right]\]

This is always $\geq 0$ (it’s a Bregman divergence), has lower variance than the naive estimator, and is what verl uses when actor.kl_loss_type=low_var_kl — implemented in seven lines at core_algos.py:2177–2183 (snippet in the code tour below). The full GRPO actor loss in verl is:

\[\mathcal{L}^{\text{actor}}_{\text{verl}}(\theta) \;=\; -\mathcal{L}^{\text{GRPO}}(\theta) \;+\; \beta \cdot \widehat{\text{KL}}_{\text{K3}}(\theta)\]

where $\beta$ is actor.kl_loss_coef (typical value: 1e-3 to 1e-4; DeepSeekMath used 0.04 for math, which is unusually high). A subtle config gotcha specific to verl: algorithm.kl_ctrl.kl_coef looks like the same thing but only takes effect when you use it as a reward shaping term (via use_kl_in_reward=True); the binding KL term in the actor loss is gated by actor.use_kl_loss=True and uses actor.kl_loss_coef. Setting one without the other does nothing.

Where the math lives in verl: a code tour

Three formulas drive the whole algorithm — the group-mean advantage $A_i$, the importance ratio $r_t$, and the K3 KL $\widehat{\text{KL}}_{\text{K3}}$ — and each is a handful of lines in verl. This section maps each formula to the exact code that produces it, so you can read along in core_algos.py and workers/utils/losses.py.

The dispatcher: which advantage estimator runs

RayPPOTrainer.fit() doesn’t know about GRPO directly — it calls a single compute_advantage() shim at ray_trainer.py:185–246 that switches on algorithm.adv_estimator. The GRPO branch is the simplest of all:

# verl/trainer/ppo/ray_trainer.py:233-245
elif adv_estimator == AdvantageEstimator.GRPO:
    grpo_calculation_mask = data.batch["response_mask"]
    advantages, returns = core_algos.compute_grpo_outcome_advantage(
        token_level_rewards=data.batch["token_level_rewards"],
        response_mask=grpo_calculation_mask,
        index=data.non_tensor_batch["uid"],
        norm_adv_by_std_in_grpo=norm_adv_by_std_in_grpo,
    )
    data.batch["advantages"] = advantages
    data.batch["returns"] = returns

Two things to notice. First, index=data.non_tensor_batch["uid"] is the per-rollout prompt UID — verl tags each of the $n$ rollouts of a prompt with the same uid so the advantage function can group them. Second, for GRPO the returns are set equal to the advantages (the function returns scores, scores at core_algos.py:331) — there’s no critic, no GAE bootstrap, so “returns” is just a name kept for API compatibility with the GAE path.

Other estimators (grpo_vectorized, gdpo, grpo_passk, dr_grpo, reinforce_plus_plus, …) all live in the same registry and are dispatched via the @register_adv_est(...) decorator at core_algos.py:116 — swapping advantage estimators is a one-line config change because they all share this interface.

Advantage: where $A_i = (R_i - \bar{R})/\sigma_R$ becomes Python

The heart of GRPO is at core_algos.py:267–331. Stripped of comments:

# verl/trainer/ppo/core_algos.py:304-329
scores = token_level_rewards.sum(dim=-1)                       # (1) per-rollout scalar R_i

id2score = defaultdict(list)
id2mean, id2std = {}, {}

with torch.no_grad():
    bsz = scores.shape[0]
    for i in range(bsz):
        id2score[index[i]].append(scores[i])                   # (2) group rollouts by uid
    for idx in id2score:
        if len(id2score[idx]) == 1:
            id2mean[idx] = torch.tensor(0.0)                   # n=1: advantage is identically 0
            id2std[idx] = torch.tensor(1.0)
        elif len(id2score[idx]) > 1:
            scores_tensor = torch.stack(id2score[idx])
            id2mean[idx] = torch.mean(scores_tensor)           # (3) bar(R)_q
            id2std[idx] = torch.std(scores_tensor)             # (4) sigma_R
    for i in range(bsz):
        if norm_adv_by_std_in_grpo:
            scores[i] = (scores[i] - id2mean[index[i]]) / (id2std[index[i]] + epsilon)  # (5) GRPO
        else:
            scores[i] = scores[i] - id2mean[index[i]]                                    # (5') Dr-GRPO
    scores = scores.unsqueeze(-1) * response_mask              # (6) broadcast to tokens

Step-by-step mapping back to the math:

1. (1) scores = token_level_rewards.sum(dim=-1) — collapses the per-token reward tensor of shape (bs, response_length) to a per-rollout scalar $R_i$. For outcome-only rewards (what GRPO is designed for) only the EOS token has nonzero reward, so this sum is effectively “pick the final reward.”
2. (2) Bucket the rollouts by their prompt UID. After this loop id2score[uid] holds the $n$ scalars ${R_1, \dots, R_n}$ that were sampled from the same prompt.
3. (3), (4) $\bar{R}$ and $\sigma_R$ per group — these are the prompt-conditional baseline and scale. They depend only on the prompt, which is why this is unbiased (see §”Why dropping the critic works”).
4. (5) The actual advantage formula — (R_i − bar(R)) / (sigma_R + eps) with norm_adv_by_std_in_grpo=True (vanilla GRPO), or just R_i − bar(R) with False (Dr-GRPO’s de-biased variant, Liu et al. 2024). The single if is the entire algorithmic difference between the two.

5. (6) Token broadcast — unsqueeze(-1) makes the scalar into a (bs, 1) tensor, then * response_mask (shape (bs, response_length)) broadcasts it across the response tokens and zeros out padding. This is what produces the token-level advantage tensor $A_{i,t} = A_i \cdot \mathbb{1}[t \le

   o_i

   ]$ that the PPO loss consumes.

The edge case at $n=1$ (id2mean=0, id2std=1) is what gives the “$n \geq 2$ is hard-required” property: with one rollout the per-token advantage is literally zero everywhere, so the policy gradient term vanishes.

Ratio: where $r_t(\theta) = \pi_\theta / \pi_{\theta_{\text{old}}}$ becomes four lines

The importance ratio is computed inside the policy loss at core_algos.py:1329–1332 — the same four lines for every PPO/GRPO variant (compute_policy_loss_vanilla, compute_policy_loss_gspo, compute_policy_loss_dppo_*, …):

# verl/trainer/ppo/core_algos.py:1329-1333
negative_approx_kl = log_prob - old_log_prob                   # log π_θ - log π_θold
negative_approx_kl = torch.clamp(negative_approx_kl, min=-20.0, max=20.0)  # exp stability
ratio = torch.exp(negative_approx_kl)                          # r_t(θ) = exp(log π_θ - log π_θold)
ppo_kl = verl_F.masked_mean(-negative_approx_kl, response_mask)  # bookkeeping

That’s it — the ratio is exp(log_prob − old_log_prob), the obvious thing. The interesting parts are where the two log-probs come from:

• log_prob (current policy) is produced by the live actor forward pass that this loss is computing gradients through. In verl’s data flow it’s model_output["log_probs"] at workers/utils/losses.py:59 — a fresh forward over the rollouts under the current weights $\theta$.
• old_log_prob (rollout-time weights) is data["old_log_probs"], materialized before the PPO inner-epoch loop by the old_log_prob phase in the trainer fit loop. This is what your perf log calls timing_s/old_log_prob: the actor (in eval mode, no gradients) replays the rollouts and stores their log-probs. Those frozen log-probs $\log \pi_{\theta_{\text{old}}}$ are what stay valid across multiple PPO epochs (ppo_epochs > 1) — every gradient pass uses the same old_log_prob and a fresh log_prob, which is exactly what makes the ratio an off-policy correction.

Three forward passes, not two — and why old_log_prob is recomputed

It’s worth being precise here, because it’s the single most-misunderstood part of the data flow: the ratio is built from two forward passes of the training engine, and the rollout that generated the tokens is a third, separate pass on a different engine. Trace the actor weights $\theta_\text{old}$ through one step:

1. Rollout (vLLM, sampling). generate_sequences() produces the tokens. vLLM can hand back its own per-token log-probs (rollout_log_probs), computed with PagedAttention kernels and the rollout dtype.
2. old_log_prob recompute (FSDP, eval, no grad). _compute_old_log_prob() at ray_trainer.py:1256 runs the training engine forward over those same tokens — output = self.actor_rollout_wg.compute_log_prob(batch_td) — to get $\log \pi_{\theta_{\text{old}}}$. Same weights as the rollout, different engine.
3. update_actor (FSDP, train, with grad). _update_actor() at ray_trainer.py:1293 does the gradient-carrying forward → $\log \pi_\theta$. The ratio in the loss is $\exp(\log\pi_\theta - \log\pi_{\theta_\text{old}})$.

So why not skip step 2 and use vLLM’s rollout_log_probs directly as the anchor? Because the vLLM forward (step 1) and the FSDP forward (step 3) are numerically different — different kernels, different dtype, different attention implementation — so vLLM’s log-probs and the trainer’s log-probs disagree by a small amount (the “rollout–training mismatch”). If you anchored the ratio to vLLM, then even at the very first gradient pass — before $\theta$ has moved at all — the ratio would be $\ne 1$, injecting a spurious off-policy correction. Recomputing $\log\pi_{\theta_\text{old}}$ with the same engine that step 3 uses guarantees the ratio is exactly 1 at the first inner step and only departs from 1 as $\theta$ actually updates. That’s the whole point of the recompute.

The pinned commit makes the choice an explicit config switch in fit() (ray_trainer.py:1527–1543):

• Decoupled mode (default) — recompute old_log_probs via step 2. Three distinct policies in play: $\pi_\text{rollout}$ (vLLM), $\pi_\text{old}$ (FSDP eval anchor), $\pi_\theta$ (FSDP train). $\pi_\text{old}$ is computed once per data batch and reused as the stable proximal anchor across all mini-batch / PPO-epoch updates.
• Bypass mode (algorithm.rollout_correction.bypass_mode=True) — skip step 2 and set old_log_probs = rollout_log_probs. Only two policies: $\pi_\text{rollout}$ and $\pi_\theta$. Cheaper (no recompute pass) but the ratio is no longer 1 at the first step, so it’s typically paired with importance-sampling correction.

When rollout_log_probs are present, verl logs the discrepancy between them and the recomputed old_log_probs (via calculate_debug_metrics) — that’s the rollout_probs_diff family in the logs, and a good health check: a large gap means your rollout and training numerics have drifted apart.

Once the ratio is in hand, the clipped surrogate is six lines at core_algos.py:1335–1354 — a direct transliteration of $\min(r_t A_t, \text{clip}(r_t, 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}) A_t)$:

# verl/trainer/ppo/core_algos.py:1335-1346
pg_losses1 = -advantages * ratio                               # -r * A
pg_losses2 = -advantages * torch.clamp(ratio, 1 - cliprange_low, 1 + cliprange_high)
clip_pg_losses1 = torch.maximum(pg_losses1, pg_losses2)        # max(-r*A, -clip(r)*A)  ≡ -min(r*A, clip(r)*A)
pg_clipfrac = verl_F.masked_mean(torch.gt(pg_losses2, pg_losses1).float(), response_mask)

Two practical notes for tuning:

• Asymmetric clip (cliprange_low ≠ cliprange_high) is the DAPO-style “clip-higher” trick: allow the policy to grow probability on good actions more aggressively ($1+\varepsilon_{\text{high}}$, e.g. 0.28) than it can shrink ($1-\varepsilon_{\text{low}}$, e.g. 0.2). This combats the entropy collapse that vanilla PPO suffers in long-horizon LM settings. In the perf doc you’ll see this as clip_ratio_low=0.2, clip_ratio_high=0.28.
• pg_clipfrac (logged as actor/pg_clipfrac) is the fraction of tokens whose ratio actually hit the clip boundary. Healthy GRPO runs sit around 0.1–0.3; if pg_clipfrac → 0 you’re under-stepping (clip never binds, KL anchor likely dominating); if → 1 your LR or batch is wrong.

KL: where $\widehat{\text{KL}}_{\text{K3}}$ becomes six lines

The K3 estimator (Schulman’s low-variance KL) is at core_algos.py:2177–2183, inside kl_penalty_forward():

# verl/trainer/ppo/core_algos.py:2177-2183
if kl_penalty in ("low_var_kl", "k3"):
    kl = ref_logprob - logprob                                 # log π_ref - log π_θ
    kl = torch.clamp(kl, min=-20, max=20)
    ratio = torch.exp(kl)                                      # π_ref / π_θ
    kld = (ratio - kl - 1).contiguous()                        # exp(x) - x - 1, Bregman ≥ 0
    return torch.clamp(kld, min=-10, max=10)

The dispatching wrapper kl_penalty() at L2126 lets you ask for "kl"/"k1" (naive log-ratio), "abs", "mse"/"k2", "low_var_kl"/"k3", or any of these with a "+" suffix (e.g. "k3+") which uses a straight-through trick to mix the K3 forward value with the K2 gradient — a small but important fix because K3’s expectation matches true KL but its gradient doesn’t (K2’s does). For most production GRPO runs k3 is fine; k3+ is worth trying if you observe KL drift that doesn’t track the loss.

Putting it together: ppo_loss() is the one function that calls them all

The orchestrator is ppo_loss() at verl/workers/utils/losses.py:57–144. Boiled down to the essentials:

# verl/workers/utils/losses.py:57-144 (essential lines)
def ppo_loss(config, model_output, data, dp_group=None):
    log_prob = model_output["log_probs"]                       # current policy log-probs
    old_log_prob = data["old_log_probs"]                       # rollout-time log-probs (frozen)
    advantages = data["advantages"]                            # from compute_grpo_outcome_advantage
    response_mask = data["response_mask"].to(bool)

    policy_loss_fn = get_policy_loss_fn(config.policy_loss.get("loss_mode", "vanilla"))
    pg_loss, pg_metrics = policy_loss_fn(                      # ratio + clipped surrogate
        old_log_prob=old_log_prob, log_prob=log_prob,
        advantages=advantages, response_mask=response_mask,
        config=config, ...,
    )

    policy_loss = pg_loss
    if config.use_kl_loss:                                     # actor.use_kl_loss=True gates K3
        ref_log_prob = data["ref_log_prob"]
        kld = kl_penalty(logprob=log_prob, ref_logprob=ref_log_prob,
                         kl_penalty=config.kl_loss_type)       # "low_var_kl" → K3
        kl_loss = agg_loss(loss_mat=kld, loss_mask=response_mask, ...)
        policy_loss += kl_loss * config.kl_loss_coef           # β · K3 added to surrogate

    return policy_loss, metrics

This is the single source of truth that ties the four pieces together — advantage from the rewards, ratio from log-probs, clipped surrogate, K3 KL anchor. Every actor backend (FSDP, Megatron, the engine workers) eventually routes its loss through this function, which is why this one 87-line file is the right place to start if you want to modify the GRPO objective.

A useful exercise for cementing the picture: trace one prompt through the system. (i) Rollouts produced by vLLM. (ii) Their token rewards land in token_level_rewards. (iii) compute_grpo_outcome_advantage() groups by uid, centers, normalizes, broadcasts → fills data["advantages"]. (iv) Actor (no gradients) runs a forward pass over the rollouts → fills data["old_log_probs"]. (v) Reference policy (no gradients) does the same → fills data["ref_log_prob"]. (vi) For each PPO inner epoch, a new actor forward (with gradients) produces model_output["log_probs"] → ppo_loss() computes ratio, clipped surrogate, K3 KL → backward. Same old_log_probs and ref_log_prob are reused across the inner epochs; only log_prob changes step to step.

What about Dr-GRPO, DAPO, GRPO++?

A taxonomy is beyond this post, but the variants all tweak the same two pieces:

• The advantage estimator. Dr-GRPO (Liu et al. 2024) removes the std normalization and the per-token loss aggregation, arguing both inject bias. DAPO (Yu et al. 2024) introduces dynamic sampling, clip-higher, and overlong reward shaping. RLOO (Ahmadian et al. 2024) uses leave-one-out group means.
• The KL anchor. Some recipes drop KL entirely and rely on the reward model + format/length penalties to stop drift; others use forward vs. reverse KL.

All of these live as algorithm.adv_estimator=* or actor.kl_loss_type=* choices in verl, with the math implemented in core_algos.py. Read that one file and you’ve read 80% of the algorithmic landscape.

verl: the architecture that runs the math

GRPO is a few-line algorithm. Running it at scale on LLMs is not. A 1.7B-actor GRPO run with a 32B-parameter reward model is ~10× the engineering complexity of pure SFT. You need:

• An actor that simultaneously trains under FSDP/Megatron and generates rollouts via vLLM (the same weights must serve both backends, fast).
• A reward model that scores rollouts at high throughput — typically a separate vLLM fleet exposed via HTTP because the reward is itself a large LM judging the actor’s outputs.
• A reference policy that’s the SFT actor at step 0, parked on CPU/disk until needed for KL.
• Ray placement groups so all of these fit on the same physical cluster without stepping on each other.

verl is one of a handful of open-source frameworks that does this well. Here’s how it carves up a cluster.

The whole launch, in one tree. From the one-line command, main_ppo.py’s TaskRunner registers roles and pools, then RayPPOTrainer.init_workers() materializes them — every call down to the literal .remote() that pins a worker to a GPU:

python -m verl.trainer.main_ppo ...
└─ TaskRunner.run()                                   main_ppo.py:223
   ├─ add_actor_rollout_worker()                      main_ppo.py:126    # actor role → class, → global_pool
   ├─ add_reward_model_resource_pool()                main_ppo.py:193    # reward → pool only, NO worker class
   ├─ init_resource_pool_mgr()                        main_ppo.py:158    # {pool: [gpus/node]} spec
   └─ RayPPOTrainer.init_workers()                    ray_trainer.py:775
      ├─ create_resource_pool() → get_placement_groups   base.py:192 / 130  # reserve GPU bundles, STRICT_PACK
      ├─ create_colocated_worker_cls()                base.py:986        # fuse actor+rollout+ref → one WorkerDict
      ├─ RayWorkerGroup(...) → cls.options(...).remote()  base.py:413    # launch 1 ActorRolloutRefWorker / GPU
      └─ actor_rollout_wg.init_model()                ray_trainer.py:895 # load FSDP shards + build vLLM engine
   # reward vLLM servers are launched separately by RewardModelManager — see the Reward subsection

The whole thing is deferral: each layer records a plan and pushes the actual GPU commitment one step later, so the single cls.options(...).remote() at the bottom places a worker with its rank, GPU, and peers all already decided (the worker reads WORLD_SIZE/RANK/MASTER_ADDR from env and runs init_process_group() itself — verl is its own torchrun). create_colocated_worker_cls is the object-level half of colocation: it dynamically builds a WorkerDict that contains an instance of every role on the pool (actor, rollout, ref — or actor/rollout/reward in the FSDP recipe), method-prefixed so the driver calls each role’s API on the same underlying Ray actor — three Python objects, one process, one CUDA context. The two subsections below are the two pools this produces.

Resource pools: actor and reward live separately

When the user sets reward.reward_model.enable_resource_pool=True, verl creates two Ray placement groups:

Pool	Sized by	Holds
global_pool	[trainer.n_gpus_per_node] * trainer.nnodes	The actor + reference policy. FSDP shards for training, vLLM engine for rollout (same weights via hybrid_engine).
reward_pool	[reward.reward_model.n_gpus_per_node] * reward.reward_model.nnodes	One or more vLLM replicas of the reward model, each serving an OpenAI-compatible /v1/chat/completions HTTP endpoint.

For a 32-GPU cluster, a typical split is 8 actor GPUs and 24 reward GPUs (e.g., a 32B reward at TP=2 → 12 replicas across 24 GPUs). The split is decided in verl/trainer/main_ppo.py (init_resource_pool_mgr builds the {pool: [gpus/node]} spec) and reserved by the ResourcePoolManager. The how — placement groups, STRICT_PACK, the per-bundle launch — is in the “Actor” subsection below.

Actor: hybrid_engine = FSDP + vLLM on the same weights

The same Qwen3-1.7B (say) parameters must be:

• Sharded under FSDP for training (gradient accumulation, optimizer offload, communication overlap).
• Materialized in vLLM for fast batched rollout generation (PagedAttention, prefix caching, async scheduling).

verl’s hybrid_engine=True mode keeps both representations resident on the same GPU. Between training steps, the actor offloads FSDP optimizer state to CPU (fsdp_config.optimizer_offload=True), wakes up the vLLM engine, runs the rollout, then puts vLLM back to sleep and pulls the optimizer state back to GPU for the gradient step. This is implemented in verl/workers/fsdp_workers.py via the ActorRolloutRefWorker / AsyncActorRolloutRefWorker classes.

A useful mental model: in hybrid_engine mode, the actor’s GPUs are doing three different jobs in sequence — generating with vLLM, scoring rewards (idle on actor GPUs; reward fleet works), then training under FSDP. They are never doing two of these at once on the same physical GPU. This is why you see GPU utilization oscillate during a GRPO run, even when nothing is wrong.

How the actor workers actually come alive. The topology is built by RayPPOTrainer.init_workers() in three phases. First it reserves GPUs — create_resource_pool() builds Ray placement groups (one bundle = one GPU, STRICT_PACK onto a single node). Then it buckets the worker class: the actor role — registered as Role.ActorRolloutRef on global_pool at main_ppo.py:140 — is wrapped in a RayClassWithInitArgs and filed under its pool (ray_trainer.py:787). Finally the spawn loop (ray_trainer.py:861): create_colocated_worker_cls fuses every role on the pool into one WorkerDict class (base.py:986), and constructing RayWorkerGroup(...) runs the loop that issues the literal cls.options(...).remote() — one ActorRolloutRefWorker per GPU bundle, with WORLD_SIZE/RANK/MASTER_ADDR injected (base.py:630) so each calls init_process_group() itself (verl is its own torchrun). Only then does actor_rollout_wg.init_model() load the FSDP shards and build the vLLM engine on every worker — created last so vLLM’s KV-cache estimate sees the FSDP allocations already in place.

Reward: HTTP, not direct call

When reward.reward_model.rollout.name=vllm and enable_resource_pool=True, verl spins up the reward-model replicas as vLLM async servers (see verl/workers/rollout/vllm_rollout/vllm_async_server.py). Each exposes /v1/chat/completions. A small router (verl provides naive_router.py in experimental/reward_loop/router/) load-balances requests across replicas.

Why HTTP and not direct Python calls? Three reasons:

1. Async-from-the-actor-side. The reward call is wrapped by asyncio in user code, which means the actor can prefill more rollouts while previous rollouts are still being scored.
2. Decoupling. The reward fleet is a separate vLLM process; if it crashes, the actor pods don’t go down with it.
3. Externalization. You can run the reward fleet as a long-lived service (LinkedIn’s “offspring” or any K8s deployment) and point many training runs at it, amortizing the cost of standing up a 32B-parameter judge.

User-side reward managers subclass RewardManagerBase and override run_single(), where the actual aiohttp POST happens. Two RM calls (one for apply-prediction, one for quality) running in parallel via asyncio.gather is the canonical pattern.

How the reward workers are initialized — and why it isn’t a WorkerGroup. The reward model is the one role registered as a pool, not a worker: add_reward_model_resource_pool() gives Role.RewardModel an entry in the pool mapping (so its GPUs are reserved) but deliberately adds no role_worker_mapping entry — so the §”actor init” spawn loop above skips it entirely; verl never drives the judge by RPC. The servers are stood up separately by the RewardModelManager: it builds a RolloutReplica per reward replica (is_reward_model=True) and calls init_colocated(resource_pool) (or init_standalone for a dedicated reward_pool). That chains to launch_servers() → server.launch_server.remote(...) → vLLMHttpServer.launch_server, which builds the AsyncLLM engine and the uvicorn /v1/... endpoint inside each server actor. The first line of launch_servers is the colocation gate — assert len(self.workers) == self.world_size, where world_size = tp × dp × pp. Because a colocated judge is sized to exactly its tensor-parallel group, it can be TP-sharded (and EP-overlaid) but not data-parallel: TP8 on 8 GPUs passes (8 == 8), but ask for TP4/DP2 and it dies with worker number 4 not equal to world size 8 before serving a token. (The actor’s own rollout vLLM servers take this identical path, via init_hybrid instead of init_colocated.)

The fit() loop

RayPPOTrainer.fit() at ray_trainer.py:1362 is the main RL loop:

# pseudocode of the real loop
for epoch in range(total_epochs):
    for batch in train_dataloader:                                # batch of prompts
        rollouts = actor.generate_sequences(batch, n=rollout_n)   # (1) actor vLLM rollout
        rewards  = reward_manager.compute_reward(rollouts)        # HTTP to reward fleet
        old_logps = actor.compute_log_prob(rollouts)              # (2) FSDP recompute, eval/no-grad → π_old anchor
        ref_logps = ref_policy.compute_log_probs(rollouts)        # for KL
        advantages = compute_advantage(rewards, ...)              # GRPO group-mean
        for ppo_epoch in range(ppo_epochs):                       # PPO inner epochs
            actor.update(rollouts, advantages, old_logps, ref_logps)  # (3) FSDP backward; ratio = exp(logπ_θ − old_logps)
        actor.update_weights_to_rollout()                         # vLLM weight sync
        if step % test_freq == 0:
            validate(actor, val_dataloader)
        if step % save_freq == 0:
            actor.save_checkpoint()

Every line maps to a verl helper. Note steps (1), (2), and (3): the rollout (vLLM), the old_log_prob recompute (FSDP eval, no grad), and the actor update (FSDP train, with grad) are three separate forward passes — and the new/old ratio is built from (2) and (3), not from the vLLM rollout (see §”Three forward passes”). The interesting work is in compute_advantage (GRPO formula) and actor.update (PPO clip with K3 KL), both already linked above.

Where each phase is actually called. Those pseudocode lines are concrete methods in fit() — and the marked_timer(...) name wrapping each is exactly the timing_s/<phase> key you see in the perf log:

Phase (timing_s/…)	Called in fit()	Delegates to
gen	async_rollout_manager.generate_sequences()	AgentLoopManager → the rollout vLLM HTTP servers
reward	_compute_reward_colocate()	→ reward_loop_manager.compute_rm_score() → HTTP to the reward fleet
old_log_prob	_compute_old_log_prob()	→ actor_rollout_wg.compute_log_prob() (FSDP, eval, no grad) → $\pi_{\text{old}}$
ref	_compute_ref_log_prob()	→ ref_policy_wg.compute_ref_log_prob()
adv	compute_advantage()	→ compute_grpo_outcome_advantage() (the §”code tour” math)
update_actor	_update_actor()	→ actor_rollout_wg.update_actor() (FSDP, train, with grad) → $\pi_\theta$, backward
update_weights	marked_timer("update_weights")	FSDP shards → vLLM engine sync, for the next rollout

One subtlety decides whether reward shows up as “free” or as a real chunk of the step: enable_agent_reward_loop = not use_rm or reward_model.enable_resource_pool. When the judge has its own resource pool, its worker handles are handed to the AgentLoopManager (:951) so scoring streams alongside rollout — that’s the <1 s reward line in the step anatomy. When the reward is colocated on global_pool (enable_resource_pool=False), there is no overlap: _compute_reward_colocate runs serially after gen, and reward becomes one of the largest phases of the step.

Anatomy of one GRPO step

To make this concrete, here’s the timing breakdown from a real single-node smoke test of a Qwen3-1.7B actor + Qwen3-32B reward model on 8 H100s (train_batch_size=64, rollout.n=2, ppo_epochs=1):

Phase	Wall time	What happens
gen	113.4 s	Actor’s vLLM generates 64 × 2 = 128 rollouts (response length up to 256 tokens)
reward	<1 s observed	Reward HTTP calls overlap with gen via async — the visible cost is just Python overhead
old_log_prob	6.0 s	Actor’s FSDP training engine (eval, no grad) re-computes log-probs of the rollouts → the $\pi_\text{old}$ anchor for the PPO ratio. Not vLLM — see §”Three forward passes”
ref	4.2 s	Reference policy log-probs (for KL)
adv	0.02 s	GRPO advantage from the 128 rewards
update_actor	19.4 s	PPO backward + FSDP all-gather/reduce + optimizer step
update_weights	4.1 s	FSDP shards → vLLM engine weight sync (for the next rollout)
step total	~147 s	One GRPO update

A few observations worth internalizing:

• Rollout dominates. ~77% of the step is gen. Bigger rollout.n and longer max_response_length push this further. This is why async rollout (rollout.mode=async) is on by default — it overlaps generation with scoring.

• Reward is “free” in the timing log but real in cost. The 32B-parameter Qwen3 judge is running in parallel with generation, on its own 6 GPUs. The wall-clock reward time displayed is just the synchronous handoff overhead.

• Slowest-rollout tail. The mean rollout time in that step was 24 s; the slowest was 112 s. Async helps but doesn’t eliminate it — one unlucky decode that hits max_response_length=256 holds the batch back. This is the single biggest tuning lever for GRPO throughput.

• The actor train step is cheap compared to rollout. 19 s for a full PPO backward over 128 rollouts (256 tokens each, 1.7B actor) is fast. If you increase the actor size to 7B-32B, this flips and update_actor becomes dominant.

Why GRPO is winning post-R1

The DeepSeek-R1 paper (January 2025) showed that GRPO + an outcome-based reward (correct/incorrect on a verifier) is sufficient to elicit long-form reasoning (“chain of thought”) from a base LLM, without any SFT on reasoning traces. That result reframed RL for LLMs from “fine-tune SFT models toward human preference” (PPO+RM) into “elicit emergent reasoning from base models” (GRPO+verifier). The compute savings from dropping the critic became the difference between a one-month research cycle and a one-week one.

Practically, this means:

• Verifiable-reward tasks (math, code, formatted outputs) are where GRPO has the strongest signal. The reward is binary, the group baseline is informative, and dense per-token credit assignment isn’t needed.
• Subjective tasks (helpfulness, harmlessness) are still PPO+RM territory in production — though GRPO with an LLM-as-judge is increasingly tried.
• Long-horizon tasks (agents, multi-turn) sit in between. The group baseline becomes noisier as the trajectory length grows, and dense credit assignment matters more.

Where to read next

• DeepSeekMath (arXiv:2402.03300) — the GRPO paper. §4 has the derivation. The appendix has the ablations vs. PPO.
• DeepSeek-R1 (arXiv:2501.12948) — GRPO at scale; reasoning emergence.
• Schulman’s KL approximation note (joschu.net) — why K3 is the right estimator. Two pages.
• PPO (arXiv:1707.06347) — the parent algorithm. §3 has the clip derivation.
• verl-project/verl — the framework. Start with verl/trainer/main_ppo.py and verl/trainer/ppo/ray_trainer.py:fit. The whole training loop fits in your head once you’ve read those two files.
• TRL’s GRPOTrainer (huggingface/trl) — a simpler implementation than verl’s, useful for cross-reference.

The math is small. The engineering is not. That’s the lesson of GRPO.