Context and the KV cache
Why a longer context costs memory, and what KV-cache quantization trades.
"Context" is how much text a model can attend to at once — your prompt plus everything generated so far. A larger context lets the model work with more input, but it isn't free: it consumes memory through the KV cache.
What the KV cache is
As a transformer processes tokens, it computes a key and a value vector for each token at each layer, and stores them so it doesn't have to recompute them for every new token. That store is the KV cache. It grows with:
- the number of tokens currently in context,
- the number of layers and attention heads in the model,
- the precision the cache is stored at.
Because it scales with context length, doubling your context roughly doubles the KV-cache memory. For long contexts on a large model, the cache can rival or exceed the size of the weights themselves — which is why a model that loads fine at short context can run out of memory at long context.
Architecture matters
Modern models often use grouped-query attention (GQA), where many query heads share a smaller number of key/value heads. Fewer KV heads means a smaller cache, which is a large part of why recent models handle long context more affordably than older ones did.
KV-cache quantization
Just as weights can be quantized, the KV cache can be stored at lower precision — commonly 8-bit or 4-bit instead of 16-bit. This directly reduces the memory a long context needs.
The trade-off:
- FP16 KV cache — the default; highest fidelity, largest footprint.
- 8-bit KV cache — roughly halves cache memory, with a small quality impact that is negligible for most uses.
- 4-bit KV cache — the smallest footprint, letting you push context further, but with a more noticeable effect on quality and a larger impact on some tasks than others.
Practical takeaways
- If you don't need a huge context, using a smaller one is the cheapest way to save memory.
- If you need long context and are memory-constrained, an 8-bit KV cache is often a good first step before dropping to 4-bit.
- Context length and KV-cache precision are both inputs to the calculator, so you can see how they change the memory budget for a given model and hardware.