Context and the KV cache

Why a longer context costs memory, and what KV-cache quantization trades.

4 min read Reviewed July 2026

TL;DR

"Context" is how much text a model can attend to at once — your prompt plus everything generated so far — and it costs memory through the KV cache. The cache grows with the number of tokens in context: doubling your context roughly doubles it, and at long contexts it can rival or exceed the weights themselves. If memory is tight, use a smaller context first, then store the cache at 8-bit (a small quality cost, negligible for most uses) before dropping to 4-bit.

What is the KV cache?

As a transformer processes tokens, it computes a key and a value vector for each token at each layer, and stores them so it doesn't have to recompute them for every new token. That store is the KV cache, and it grows with:

  • the number of tokens currently in context,
  • the number of layers and attention heads in the model,
  • the precision the cache is stored at.
Weights (fixed) same at any context KV cache grows with context 4K 8K 16K 32K
The weights are a fixed cost paid at load; the KV cache is a running cost that grows with every token in context.

The weights are a one-off bill. The cache is a metered one: because it scales with context length, doubling your context roughly doubles the KV-cache memory. For long contexts on a large model, the cache can rival or exceed the size of the weights themselves — which is why a model that loads fine at short context can run out of memory at long context, and why the fit estimates in the VRAM guide always assume a stated context window.

To get a feel for the scale of it, drag the slider — it shows how the cache claims more of the budget as you push the context out:

Try it: KV-cache size vs context

8B-class model, grouped-query attention
Context length
KV cache
KV-cache precision

Doubling the context doubles the cache; dropping fp16 → Q8 → Q4 cuts it roughly in half each step — not quite exactly, because a quantised cache stores 32 values per block and keeps a 16-bit scale alongside them. This reference model spends about 128 KiB per token at fp16. Figures are precomputed from the same KV-cache formula the fit engine and calculator use; your own model's numbers scale with its layer and KV-head counts.

Why do newer models handle long context better?

Modern models often use grouped-query attention (GQA), where many query heads share a smaller number of key/value heads. The cache stores keys and values — not queries — so fewer KV heads means a smaller cache at any given context length. This is a large part of why recent models handle long context more affordably than older ones did: the architecture changed, not just the memory on the card. It's also why two models of a similar parameter count can show quite different memory behaviour once the context grows — the KV-head count matters as much as the size on disk.

Can the cache itself be quantized?

Yes. Just as weights can be quantized, the KV cache can be stored at lower precision — commonly 8-bit or 4-bit instead of 16-bit. It's a separate dial from the weight quant, and it directly reduces the memory a long context needs.

Cache precision Footprint Quality note
FP16 Largest — the default Highest fidelity; the reference point
8-bit Roughly half of FP16 Small impact, negligible for most uses
4-bit Smallest — pushes context furthest More noticeable, and hits some tasks harder than others

The asymmetry is worth noting: the step from FP16 to 8-bit buys a large saving for a cost most users never see, while the step from 8-bit to 4-bit buys less additional room for a more visible cost. That behaviour is why 8-bit is the sensible first move rather than jumping straight to the smallest cache.

What should you actually do?

If context is squeezing your memory

  1. Use a smaller context. If you don't need a huge window, this is the cheapest saving of all — the cache shrinks in direct proportion.
  2. Try an 8-bit KV cache. Roughly halves cache memory; often a good first step before touching anything else.
  3. Drop to a 4-bit cache only if you must. It buys the most room for context, but with a more noticeable effect on quality.
  4. Rebalance the whole budget. If it still doesn't fit, a smaller model or a lower weight quant frees memory that context can then use — see the VRAM guide for the full picture.

Context length and KV-cache precision are both inputs to the calculator, so you don't have to guess at any of this.