Tracing the HuggingFace Trainer Dataloader Path — From `trainer.train()` to a `next(epoch_iterator)`

A code-level walk through how the HuggingFace `Trainer` constructs and iterates the training DataLoader — sampler selection, the `accelerator.prepare(...)` wrapping, the gradient-accumulation prefe...

May 12, 2026 Machine Learning Training Framework

PyTorch DDP Deep Dive — Initialization, Buffer Broadcast, the Reducer, and the torch.compile Story

A code-level walk through `DistributedDataParallel` — `__init__`, the bucket-grad-allreduce machinery in C++ `Reducer`, the per-forward buffer broadcast that surprises everyone the first time they ...

May 11, 2026 Machine Learning Performance, Distributed Training

How vLLM's `torch.compile` Backend Handles Dynamic Batch Sizes — A Code Walkthrough

A code-level dive into vLLM's piecewise compilation strategy — how a single Dynamo FX graph captured with dynamic shapes feeds into N concrete-shape Inductor compilations, and how the runtime dispa...

May 10, 2026 Machine Learning Performance, torch.compile

Qwen3 Architecture Layer-by-Layer — Reading the HuggingFace Implementation

A code-level walk of `Qwen3ForCausalLM` end-to-end — top-level wrapper, transformer trunk, decoder block, attention with q_norm/k_norm + GQA + per-layer sliding window, SwiGLU MLP, RMSNorm, RoPE — ...

May 8, 2026 LLM Architecture

Torch.compile Deep Dive II — Inductor Codegen and Buffer Lifetimes, End to End

A code-level walk through PyTorch Inductor's compilation pipeline — from the FX graph AOTAutograd hands over, through IR lowering, the scheduler, fusion, memory planning, wrapper codegen, and Trito...

Apr 30, 2026 Machine Learning Performance, torch.compile

SGLang Deep Dive: How Engine.generate() Boots and Runs

A code-level walk through SGLang's offline engine — the three-process architecture, where the scheduler and detokenizer subprocesses are spawned, what an Engine.generate() call traverses, and why t...

Apr 28, 2026 Machine Learning Performance, LLM Inference

vLLM v1 Deep Dive: How LLM(...) and the Server Boot and Generate

A code-level walk through vLLM v1 — the six-layer architecture, where the EngineCore subprocess is spawned, what a generate() call traverses, plus the online (vllm serve) startup + request path and...

Apr 27, 2026 Machine Learning Performance, LLM Inference

CUTLASS Deep Dive: From CuTe Layouts to Blackwell tcgen05

A code-level tour of CUTLASS and CuTe — layouts, tiled copies, SM80 pipelines, Hopper WGMMA+TMA, Blackwell tcgen05 and TMEM, and advanced GEMM fusions

Apr 24, 2026 Machine Learning Performance, Kernel

NCCL Deep Dive: From P2P/CUMEM to torchrun Init

Deep dive into NCCL — what it is, how P2P/CUMEM works at the hardware level, and how torchrun bootstraps NCCL from scratch

Apr 10, 2026 Machine Learning Performance, Communication

From Softmax Attention to FlashAttention-2

Derive one-pass softmax attention and implement the same simplified FlashAttention-2 forward algorithm in Triton and CuTe DSL

Aug 14, 2025 Machine Learning Performance, Kernel