Module 6 of 6

Inference engineering: serving real users

Training happens once. Inference happens every time someone sends a message, millions of times a day. For popular models, serving can cost more than training over the model's lifetime.

Two phases of every request

Prefill: read the prompt

All prompt tokens are processed in parallel in one big pass. Lots of maths per byte, so it is compute-bound. It sets the time to first token.

Decode: write the answer

One token at a time. Each step reads every weight and the whole KV cache for very little maths, so it is memory-bandwidth-bound. It sets the time per output token.

Latency \approx TTFT + (n - 1) \times TPOT

TTFT time to first token (prefill)n output tokensTPOT time per output token (decode)

Why decode is limited by bandwidth

To produce one token for one user, the GPU must stream every weight from HBM. So the fastest possible decode speed is set by memory bandwidth, not FLOPS.

\frac{tokens}{second} \leq \frac{memory bandwidth}{bytes read per token}

e.g. 3.35 TB/s on an H100all weights + this user's KV cache

The KV cache

When generating token 500, the model needs the keys and values of the 499 tokens before it. Recomputing them every step would be wasteful, so they are stored in the KV cache. It speeds decode up, but it eats memory, and it grows with every token and every user.

KV cache = 2 \times L \times H_{kv} \times d_{head} \times S \times B \times bytes

2 one K and one VL layersH_kv key/value headsd_head head sizeS tokens so farB users in the batch

Grouped-query attention (GQA) lets several query heads share one key/value head, so the cache shrinks. Llama 3 8B uses 8 KV heads instead of 32: a 4× smaller cache for almost the same quality.

Batching: keep the GPU full

Static batching

The whole batch waits until its longest answer finishes. Short requests leave empty slots, wasting the GPU.

Continuous batching

Finished requests leave and new ones join at every decode step, so slots stay full. Used by engines like vLLM, SGLang and TensorRT-LLM.

PagedAttention (vLLM) stores the KV cache in small fixed-size blocks, like pages in an operating system. Memory isn't reserved for the longest possible answer up front, so many more users fit on one GPU.

Speculative decoding

Small draft model guesses 4 tokens

→

Big model checks all 4 in one pass

→

Keep the guesses it agrees with

Checking several tokens at once costs about the same as generating one, because decode is memory-bound. When the draft is right, you get several tokens for the price of one step, and the output follows exactly the same distribution as the big model alone.

What inference engineers optimize

Latency

TTFT and time per token for each user.

Throughput

Tokens per second across all users.

Cost

Price per million tokens.

Memory

Weights plus the KV cache for every user.

Bigger batches raise throughput but make each user wait longer. Inference engineering is choosing the right point on that trade-off, then using quantization, KV cache tricks, batching and speculation to move the whole curve.

It all connects! Decode is memory-bandwidth-bound, so anything that moves fewer bytes per token (quantization, GQA, batching) makes it faster. Same lesson as FlashAttention: move less data.

See slides 72 to 74

Previous: Multi-GPU Training Mark done and take the quiz