◫ Vault

Series · 11 of 11 published

Nano-vLLM Systems Study

A two-part systems study of nano-vLLM, from the idle-GPU mental model through the engine loop that schedules and serves requests.

  1. Unit 1 Foundations & Roadmap Part I: Foundations Published

    Build the nano-vLLM mental model: why decode starves GPUs, why batching matters, and how the inference harness eliminates idle work.
  2. Unit 2 Architecture & the Request Lifecycle Part II: Request Lifecycle Published

    Trace nano-vLLM's request lifecycle through the engine, scheduler, runner, and sequence state machine from prompt to generated text.
  3. Unit 3 Continuous Batching & the Scheduler Part III: Scheduling Published

    See how nano-vLLM keeps GPUs full with continuous batching, prefill-first scheduling, resource budgets, and safe preemption under KV-cache pressure.
  4. Unit 4 PagedAttention & the Block Manager Part IV: Memory Management Published

    Learn how nano-vLLM treats the KV cache as paged memory to eliminate fragmentation, grow sequences in O(1), and sustain larger batches.
  5. Unit 5 Prefix Caching Part V: Reuse Published

    See how nano-vLLM reuses shared KV-cache prefixes through chained block hashes, ref-counted sharing, and lazy eviction.
  6. Unit 6 Chunked Prefill Part VI: Scheduling Published

    Learn how nano-vLLM splits an over-budget prompt across correct prefill steps without generating a token until the final chunk.
  7. Unit 7 The Forward Pass & Attention on the Cache Part VII: Execution Published

    Trace ModelRunner from scheduled sequences to packed GPU tensors, paged KV-cache writes, FlashAttention, and one sampled token per sequence.
  8. Unit 8 Sizing the KV Cache & Warmup Part VIII: Memory Sizing Published

    See how nano-vLLM measures activation memory during warmup, sizes the KV-cache block pool, and wires cache views into attention layers.
  9. Unit 9 CUDA Graphs Part IX: GPU Execution Published

    Learn how nano-vLLM captures fixed-shape decode work in CUDA graphs to eliminate kernel-launch overhead while keeping prefill eager.
  10. Unit 10 Tensor Parallelism Part X: Scale Published

    See how nano-vLLM shards model weights and execution across GPUs with column/row parallel layers, all-reduce, and an SPMD runtime.
  11. Unit 11 The Small Precise Pieces Part XI: Finale Published

    Finish the study with nano-vLLM's sampler, RMSNorm, RoPE, SwiGLU, weight loading, torch.compile, and an end-to-end engine recap.

← Back to the library