PyTorch DDP Deep Dive — Initialization, Buffer Broadcast, the Reducer, and the torch.compile Story

A code-level walk through `DistributedDataParallel` — `__init__`, the bucket-grad-allreduce machinery in C++ `Reducer`, the per-forward buffer broadcast that surprises everyone the first time they ...

May 11, 2026 Machine Learning Performance, Distributed Training

How vLLM's `torch.compile` Backend Handles Dynamic Batch Sizes — A Code Walkthrough

A code-level dive into vLLM's piecewise compilation strategy — how a single Dynamo FX graph captured with dynamic shapes feeds into N concrete-shape Inductor compilations, and how the runtime dispa...

May 10, 2026 Machine Learning Performance, torch.compile

Qwen3 Architecture Layer-by-Layer — Reading the HuggingFace Implementation

A code-level walk of `Qwen3ForCausalLM` end-to-end — top-level wrapper, transformer trunk, decoder block, attention with q_norm/k_norm + GQA + per-layer sliding window, SwiGLU MLP, RMSNorm, RoPE — ...

May 8, 2026 LLM Architecture

Torch.compile Deep Dive II — Inductor Codegen and Buffer Lifetimes, End to End

A code-level walk through PyTorch Inductor's compilation pipeline — from the FX graph AOTAutograd hands over, through IR lowering, the scheduler, fusion, memory planning, wrapper codegen, and Trito...

Apr 30, 2026 Machine Learning Performance, torch.compile

SGLang Deep Dive: How Engine.generate() Boots and Runs

A code-level walk through SGLang's offline engine — the three-process architecture, where the scheduler and detokenizer subprocesses are spawned, what an Engine.generate() call traverses, and why t...

Apr 28, 2026 Machine Learning Performance, LLM Inference

vLLM v1 Deep Dive: How LLM(...) and the Server Boot and Generate

A code-level walk through vLLM v1 — the six-layer architecture, where the EngineCore subprocess is spawned, what a generate() call traverses, plus the online (vllm serve) startup + request path and...

Apr 27, 2026 Machine Learning Performance, LLM Inference

CUTLASS Deep Dive: From CuTe Layouts to Blackwell tcgen05

A code-level tour of CUTLASS and CuTe — layouts, tiled copies, SM80 pipelines, Hopper WGMMA+TMA, Blackwell tcgen05 and TMEM, and advanced GEMM fusions

Apr 24, 2026 Machine Learning Performance, Kernel

NCCL Deep Dive: From P2P/CUMEM to torchrun Init

Deep dive into NCCL — what it is, how P2P/CUMEM works at the hardware level, and how torchrun bootstraps NCCL from scratch

Apr 10, 2026 Machine Learning Performance, Communication

From FlashAttention to FlashAttention 2

Deep dive into FlashAttention 2 — algorithm, code, and why it's faster

Aug 14, 2025 Machine Learning Performance, Kernel

Torch.compile 101 - From Python Function to Triton Kernel

An interactive deep dive into PyTorch's torch.compile system, tracing the journey from Python functions to optimized Triton kernels with hands-on examples and performance analysis.

Aug 7, 2025 Machine Learning Performance, torch.compile