Bandwidth vs compute: why tokens/sec tracks memory bandwidth

When a model is writing its answer, it is limited by how fast it can read its weights from memory — not by raw compute. This is the rule of thumb behind ranking hardware by bandwidth.

When people ask "how fast will this model run?", the answer during token generation comes down to one number more than any other: memory bandwidth — how many gigabytes per second the chip can read from its memory.

Why generation is bandwidth-bound

To produce each new token, the model reads through its weights once. On a modern GPU the arithmetic is cheap and fast; the slow part is fetching those weights from VRAM. So generation speed is set by how quickly memory can be streamed, not by how many FLOPs the chip can do.

That gives a simple rule of thumb:

tokens/sec ≈ memory bandwidth ÷ bytes read per token

and the bytes read per token are roughly the size of the model's weights in memory (which depends on the quant — see Which quant should I pick? and K-quants, I-quants and imatrix).

A concrete example

Take two real cards from the roster. A used RTX 3090 has 936 GB/s of memory bandwidth; an RTX 4070 Ti has 504 GB/s. The 3090 is the older card, and on paper the 4070 Ti has newer, faster compute cores — yet for decoding tokens the 3090 is the faster card, because:

936 ÷ 504 ≈ 1.85× more bandwidth

For a ~8 GB quant, the 3090's ceiling is about 936 ÷ 8 ≈ 117 tok/s versus about 504 ÷ 8 ≈ 63 tok/s on the 4070 Ti. Those are theoretical ceilings — real throughput is a fraction of them because of attention, the KV cache, and runtime overhead — but the ratio holds, and it explains why a bandwidth-first reading of hardware often ranks an older high-bandwidth card above a newer mid-range one.

Where compute does matter

Raw compute (TFLOPS) drives prompt processing — ingesting your prompt before the first token appears — not decoding. That is a different speed with different bottlenecks; see Prompt processing vs token generation.

And for mixture-of-experts models, only the active weights are read per token, so the "bytes per token" in the formula is much smaller than the full model — see MoE vs dense.

Takeaways

  • For generation speed, compare memory bandwidth, not core counts or TFLOPS.
  • Bytes-per-token depends on your quant: a bigger quant decodes slower, a smaller one faster.
  • The rule of thumb gives a ceiling; expect real tok/s to be a share of it.