◫ Vault

Series · 3 of 5 published

Inside vLLM

A companion series to Aleksa Gordic's anatomy of vLLM — from one engine step() through speculative decoding to the distributed serving stack that turns a curl into GPU work.

  1. Unit 1 Inside one engine step Part I: Engine core Published

    How a single vLLM engine step() spends a token budget and a block pool: decode-first scheduling, chunked prefill, prefix caching, and preemption.
  2. Unit 2 Speculative decoding Part II: Advanced features Published

    Why a draft model can accelerate a large LM without changing the output distribution, and why acceptance rate and batch size set the speedup.
  3. Unit 3 From UniProc to MultiProc Part III: Scaling up Coming soon
  4. Unit 4 Distributed serving Part IV: Serving Published

    How a curl request reaches a GPU in vLLM: data-parallel replicas, the three-thread engine process, ZMQ, and the load-balancing score that picks a replica.
  5. Unit 5 Latency vs throughput Part V: Benchmarks Coming soon

← Back to the library