A code-level tour of Megatron fine-tuning: where Qwen recipes live, how pretraining and SFT datasets are built, how pretrain_gpt.py reaches the forward/backward schedule, which extension points mat...

A source-code tour of Miles' RL loop: start from train.py, follow JSONL prompts into SGLang rollout and reward scoring, see how GRPO trains on generated tokens rather than labels, and understand wh...

A deep dive into why PyTorch native C++ extensions are often tied to a specific Torch build, how TVM FFI provides a stable framework-agnostic ABI, how PyTorch tensors cross the boundary through DLP...

A deep dive into how Mixture-of-Experts models are parallelized for inference serving. What tensor (TP), expert (EP), and data (DP) parallelism each shard inside an MoE, what they do to the all-to-...

A companion to the NCCL deep dive — what NVSHMEM is, why it's a fundamentally different communication model (one-sided, device-initiated PGAS vs host-launched collectives), how it surfaces in the P...

One mental model for Mixture-of-Experts, then the same idea traced through four codebases — a plain-PyTorch reference, HuggingFace Qwen3-MoE, vLLM's fused inference kernels, and Megatron-Core's tra...

A code-level tour of what vLLM does with the single number `gpu_memory_utilization` — from the config field, through `request_memory` (it's total × util, not free × util), the three-category memory...

A first-principles derivation of GRPO (the post-DeepSeek-R1 default for LLM RL) — what it removed from PPO, why dropping the critic works, the K3 KL estimator in the actor loss, and how the open-so...

A walk through the LM-head + cross-entropy fwbw kernel — why the `(BT, V)` logits tensor is the memory wall every LLM training framework eventually hits, the softmax-minus-onehot identity that make...

A code-level walk through how the HuggingFace `Trainer` constructs and iterates the training DataLoader — sampler selection, the `accelerator.prepare(...)` wrapping, the gradient-accumulation prefe...