The vLLM Jargon I Finally Started Understanding

For a while, reading vLLM's docs felt like reading a language I half-knew. I understood the words individually and none of the sentences. This is me writing down the terms I finally stopped being scared of, in the order they clicked.

Chunked prefill

When a request comes in, the model first has to process the whole prompt before it can generate a single token. That first pass is called prefill, and for a long prompt it's expensive — you're running attention over thousands of tokens at once.

The problem is that a big prefill blocks everyone else. While the GPU is busy chewing through one 8,000-token prompt, every other request that just wants its next token is waiting.

Chunked prefill breaks that big prompt into smaller pieces and interleaves them with the decode steps of other requests. Instead of "finish this whole prefill, then serve everyone else," it's "do a chunk of prefill, serve a round of decodes, do another chunk." Latency for everyone else stops falling off a cliff whenever a long prompt shows up.

Prefix caching

This is the one that took me the longest, and the one I ended up drawing to understand.

The idea: if two requests share the same beginning — the same system prompt, the same few-shot examples — then the model's internal work on that shared part is identical. There's no reason to recompute it. Prefix caching stores that work (the key/value tensors) and reuses it.

The mechanism is a chain of hashes. The prompt is split into fixed-size blocks of tokens. Each block gets a hash, but the hash isn't just of that block's tokens — it folds in the hash of the block before it.

h0 = hash(∅,  tokens_of_block_0)
h1 = hash(h0, tokens_of_block_1)
h2 = hash(h1, tokens_of_block_2)

So a block's identity depends on everything that came before it. If two requests start the same way, they produce the same chain of hashes for as long as they agree — and every one of those blocks is a cache hit.

The thing that finally made it concrete for me: change a single token early in the prompt and every hash after it changes too. The chain is exactly why a small edit near the start invalidates the entire suffix — there's no partial credit past the point where two prompts diverge.

Guided decoding

Sometimes you don't want the model to say whatever it wants — you want valid JSON, or a value from a fixed set of options. Guided decoding constrains generation so the output is guaranteed to match a grammar or schema.

The way it works is less magical than it sounds. At each step the model produces scores over the whole vocabulary. Guided decoding masks out every token that would break the structure — if the grammar says the next character has to be a closing brace or a digit, every other token's probability gets forced to zero before sampling. You're not asking the model nicely to produce JSON; you're making invalid tokens unpickable.

Speculative decoding

Generating one token at a time is slow because each token needs a full forward pass through a huge model. Speculative decoding tries to get more than one token per expensive pass.

A small, fast "draft" model guesses the next few tokens. Then the big model checks all of those guesses in a single pass — which it can do, because verifying is cheaper than generating. Any guessed tokens the big model agrees with are accepted for free; the first one it disagrees with is where it takes over. When the draft model is right often enough, you get several tokens for roughly the cost of one.

The key insight is that it's lossless: the output distribution is still the big model's, because the big model has final say on every token. The draft is only ever a proposal.

P/D disaggregation

Prefill and decode have completely different appetites. Prefill is compute-heavy and runs over many tokens at once. Decode is memory-bandwidth-heavy and produces one token at a time. Running both phases on the same GPU means neither runs on hardware suited to it.

Prefill/decode disaggregation splits them across different machines: one pool of GPUs does nothing but prefill, another does nothing but decode, and the computed KV cache is handed from the first to the second. Each pool gets to be sized and tuned for the one thing it does.

Where this left me

None of these are the model itself — they're the systems wrapped around it to make serving it practical. That's the part I keep getting pulled toward. The model is a big function; the interesting engineering is in everything that decides when and how that function actually runs.

I'm sure I've got details wrong here, and I'll come back and fix them as I learn more. But writing them down in my own words was the point where they stopped being jargon.