Prompt processing vs token generation

Local-LLM speed is really two numbers, and confusing them leads to surprises. They are usually written pp and tg.

Token generation (tg / decode)

Token generation is the model writing its answer, one token at a time. Each step reads through the (active) weights once, so it is memory-bandwidth bound — this is the tok/s figure people usually quote, and the one the bandwidth rule of thumb predicts.

Prompt processing (pp / prefill)

Prompt processing is the model reading your prompt before it writes anything — the system prompt, the chat history, and any retrieved documents. Unlike decoding, prefill can chew through all those tokens in parallel, so it is compute (FLOPS) bound rather than bandwidth bound, and per-token it is usually much faster than generation.

But it scales with prompt length. Feed in a few thousand tokens of context and you pay a real prefill cost before the first output token appears — the "time to first token".

Why long context and RAG bottleneck on ingestion

A short chat prompt makes prefill negligible. But retrieval-augmented generation (RAG), long documents, or a large system prompt push thousands of tokens through prefill every request. At that point pp dominates the wait, and a card with strong compute and FlashAttention-class kernels pulls ahead — which is largely an NVIDIA/CUDA advantage today (see GPU software stacks). The KV cache those tokens create also grows the memory bill (Context and the KV cache).

How this site reports the two

The estimated speeds shown across the catalog are decode (tg), derived from the memory-bandwidth roofline, because decode is what that model predicts well. Prompt-processing figures are shown only where they have been measured by the community, since pp depends heavily on the backend, kernels, and hardware and is not reliably predicted by the simple roofline. When you see a pp number, read it as an observation, not an estimate.