Bandwidth vs compute: why tokens/sec tracks memory bandwidth

TL;DR

When a model is writing its answer, it re-reads its weights for every token it generates — so tokens/sec during generation is set by memory bandwidth, not by raw compute. As a rule of thumb, tokens/sec ≈ bandwidth ÷ bytes read per token, where bytes-per-token is roughly the model's size in memory. That's why a used RTX 3090 (936 GB/s) out-decodes a newer RTX 4070 Ti (504 GB/s), despite the 4070 Ti's more modern compute cores.

Why is generation bandwidth-bound?

To produce each new token, the model reads through its weights once. On a modern GPU the arithmetic is cheap and fast; the slow part is fetching those weights from VRAM. Generation speed is therefore set by how quickly memory can be streamed, not by how many FLOPs the chip can do.

That gives a simple rule of thumb:

tokens/sec ≈ memory bandwidth ÷ bytes read per token

The bytes read per token are roughly the size of the model's weights in memory, which depends on the quant — see Which quant should I pick? and K-quants, I-quants and imatrix. This is the under-appreciated half of the formula: a smaller quant isn't just a memory saving, it's fewer bytes streamed per token, so the same card decodes it faster.

A concrete example: RTX 3090 vs RTX 4070 Ti

Take two real cards from the roster. The RTX 3090 is the older card, and on paper the 4070 Ti has newer, faster compute cores — yet for decoding tokens the 3090 is the faster card, because 936 ÷ 504 ≈ 1.85× more bandwidth. Run a ~8 GB quant through the rule of thumb and the gap is plain:

Where does compute matter?

Raw compute (TFLOPS) drives prompt processing — ingesting your prompt before the first token appears — not decoding. That is a different speed with different bottlenecks; see prompt processing vs token generation. If your workload is long prompts and short answers — summarising documents, say — that compute-bound phase is the one you'll actually be waiting on.

Two other cases bend the simple formula:

• Mixture-of-experts models read only the active weights per token, so the "bytes per token" is much smaller than the full model — and decoding is correspondingly faster than the total parameter count suggests. See MoE vs dense.
• GPU+CPU offload splits the layers across two memory pools. The layers held in system RAM stream at system-RAM speeds, which drags the overall pace down — the VRAM guide covers when offloading is worth it anyway.

How to compare hardware, then

Capacity and bandwidth are different axes: VRAM size decides what fits (that's the VRAM guide's territory), while bandwidth decides how fast it runs once it does. A spec sheet leads with cores and TFLOPS; for local generation speed, read past them.

• For generation speed, compare memory bandwidth, not core counts or TFLOPS.
• Bytes-per-token depends on your quant: a bigger quant decodes slower, a smaller one faster, on the same card.
• The rule of thumb gives a ceiling; expect real tok/s to be a share of it, and use the ratio between cards rather than the absolute number.

Sources & data

• RTX 3090 and RTX 4070 Ti memory bandwidth (936 GB/s, 504 GB/s): catalog GPU roster (DR-23-verified specs)