Mixed Precision Deep Dive
A practical guide to mixed precision on NVIDIA GPUs: floating-point formats, CUDA Cores versus Tensor Cores, FP8 and NVFP4 scaling, checkpoint storage, runtime quantization, and which Transformer l...
ZCache
Megatron-LM Fine-Tuning Deep Dive: Qwen Recipes, Datasets, and the Training Loop
A code-level tour of Megatron fine-tuning: where Qwen recipes live, how pretraining and SFT datasets are built, how pretrain_gpt.py reaches the forward/backward schedule, which extension points mat...
Miles Deep Dive: Main Loop, Rollout, and Reward
A source-code tour of Miles' RL loop: start from train.py, follow JSONL prompts into SGLang rollout and reward scoring, see how GRPO trains on generated tokens rather than labels, and understand wh...
TVM FFI Deep Dive: Shipping Native Kernels Without Tying Them to One PyTorch Build
A deep dive into why PyTorch native C++ extensions are often tied to a specific Torch build, how TVM FFI provides a stable framework-agnostic ABI, how PyTorch tensors cross the boundary through DLP...
Serving MoE Models: A Deep Dive into Parallelism Strategies (TP, EP, DP)
A deep dive into how Mixture-of-Experts models are parallelized for inference serving. What tensor (TP), expert (EP), and data (DP) parallelism each shard inside an MoE, what they do to the all-to-...
NVSHMEM Deep Dive: One-Sided GPU Communication, and How It Differs from NCCL
A companion to the NCCL deep dive — what NVSHMEM is, why it's a fundamentally different communication model (one-sided, device-initiated PGAS vs host-launched collectives), how it surfaces in the P...
Mixture-of-Experts Across the Stack: From a 20-Line Reference to Wide Expert Parallelism
One mental model for Mixture-of-Experts, then the same idea traced through four codebases — a plain-PyTorch reference, HuggingFace Qwen3-MoE, vLLM's fused inference kernels, and Megatron-Core's tra...
Where gpu_memory_utilization Actually Goes: vLLM's Memory Budget, Line by Line
A code-level tour of what vLLM does with the single number `gpu_memory_utilization` — from the config field, through `request_memory` (it's total × util, not free × util), the three-category memory...
GRPO from First Principles, and How verl Implements It
A first-principles derivation of GRPO (the post-DeepSeek-R1 default for LLM RL) — what it removed from PPO, why dropping the critic works, the K3 KL estimator in the actor loss, and how the open-so...
Fused Linear Cross-Entropy: The Math, the Memory Wall, and Two Ways Around It
A walk through the LM-head + cross-entropy fwbw kernel — why the `(BT, V)` logits tensor is the memory wall every LLM training framework eventually hits, the softmax-minus-onehot identity that make...