MoE vs dense: why a 30B-A3B runs near 8B speed

Mixture-of-experts models activate only a slice of their weights per token, so they decode about as fast as a small model — but you still need memory for the whole thing.

Model names like 30B-A3B or 235B-A22B describe a mixture-of-experts (MoE) model. The two numbers matter for very different reasons: one sets the memory you need, the other sets the speed you get.

Dense vs MoE

A dense model uses every weight to produce every token. A 13B dense model reads all 13B parameters per token.

An MoE model splits much of the network into many "experts" and adds a small router that, for each token, picks only a few experts to run. The rest sit idle for that token. So only a fraction of the weights are active at any step.

The A-notation

The A number is the count of active parameters per token, in billions:

  • 30B-A3B — about 30B parameters total, but only ~3B active per token.
  • 235B-A22B — about 235B total, ~22B active per token.

Why it decodes fast

Token generation is bound by how many bytes you read per token (Bandwidth vs compute). Because an MoE only reads its active weights each token, a 30B-A3B streams roughly the bytes of a ~3B model — so it decodes at close to small-model speed, well above what a 30B dense model would manage.

Why it still needs the memory of a big model

Here is the catch: any token can route to any expert, so all the experts have to be resident in memory. You must fit the whole ~30B (at your quant) in VRAM or unified memory, even though only ~3B run per token.

MoE: memory of the big number, decode speed near the small number.

Plan your memory budget against the total size (see the VRAM guide), and your speed expectations against the active size.

Trade-offs to keep honest

  • Prompt processing touches more of the network than a single decode step, so the prefill cost is not as small as the active-parameter count alone suggests (pp vs tg).
  • MoE buys more stored knowledge per unit of compute, but a 3B-active model is not automatically as capable as a 30B-dense model on every task — treat active and total counts as two different things, not one.