ZCache

https://supercharleszhu.github.io/ZCacheA minimal, responsive and feature-rich Jekyll theme for technical writing. 2026-08-24T10:38:41+00:00 supercharles https://supercharleszhu.github.io/ Jekyll © 2026 supercharles /assets/img/favicons/favicon.ico /assets/img/favicons/favicon-96x96.png Megatron-LM Fine-Tuning Deep Dive: Qwen Recipes, Datasets, and the Training Loop2026-08-11T07:00:00+00:00 2026-08-11T07:00:00+00:00 https://supercharleszhu.github.io/posts/2026-08-11-00/ Charles Zhu

A code-level tour of Megatron fine-tuning: where Qwen recipes live, how pretraining and SFT datasets are built, how pretrain_gpt.py reaches the forward/backward schedule, which extension points matter, and a successful eight-GPU Qwen-style MCore smoke test with TP, PP, CP, and EP all set to one.

Miles Deep Dive: Main Loop, Rollout, and Reward2026-06-29T02:00:00+00:00 2026-08-11T06:26:37+00:00 https://supercharleszhu.github.io/posts/2026-06-29-00/ Charles Zhu

A source-code tour of Miles' RL loop: start from train.py, follow JSONL prompts into SGLang rollout and reward scoring, see how GRPO trains on generated tokens rather than labels, and understand where the architecture differs from verl.

TVM FFI Deep Dive: Shipping Native Kernels Without Tying Them to One PyTorch Build2026-06-26T02:00:00+00:00 2026-08-11T06:26:37+00:00 https://supercharleszhu.github.io/posts/2026-06-26-00/ Charles Zhu

A deep dive into why PyTorch native C++ extensions are often tied to a specific Torch build, how TVM FFI provides a stable framework-agnostic ABI, how PyTorch tensors cross the boundary through DLPack and TensorView, and when to choose TVM FFI versus PyTorch's stable ABI.

Serving MoE Models: A Deep Dive into Parallelism Strategies (TP, EP, DP)2026-06-08T06:00:00+00:00 2026-08-11T06:26:37+00:00 https://supercharleszhu.github.io/posts/2026-06-08-01/ Charles Zhu

A deep dive into how Mixture-of-Experts models are parallelized for inference serving. What tensor (TP), expert (EP), and data (DP) parallelism each shard inside an MoE, what they do to the all-to-all / all-reduce communication and per-GPU memory

NVSHMEM Deep Dive: One-Sided GPU Communication, and How It Differs from NCCL2026-06-08T02:00:00+00:00 2026-08-11T06:26:37+00:00 https://supercharleszhu.github.io/posts/2026-06-08-00/ Charles Zhu

A companion to the NCCL deep dive — what NVSHMEM is, why it's a fundamentally different communication model (one-sided, device-initiated PGAS vs host-launched collectives), how it surfaces in the PyTorch world via torch.distributed._symmetric_memory, its programming model (symmetric heap, put/get/signal, host vs device API), and the advanced IBGDA-vs-IBRC transport distinction that decides MoE ...