K-quants, I-quants and imatrix

What the K in Q4_K_M means, how I-quants (IQ...) differ, and what an importance matrix (imatrix) actually does during quantization.

GGUF quant names hide three different ideas: K-quants, I-quants, and imatrix. They are often mixed up, so here is what each one is.

Legacy quants

The oldest GGUF quants — Q4_0, Q4_1, Q8_0 — store weights in simple fixed-size blocks with one scale per block. They still work and Q8_0 is a fine near-lossless option, but for 4–6 bit work they have been largely superseded.

K-quants

K-quants are the _K names: Q3_K, Q4_K, Q5_K, Q6_K, each with a size class S, M, or L (small/medium/large). Instead of one uniform block scheme, they spend more bits on the tensors that matter most and fewer elsewhere, so they hold quality better per bit than the legacy quants. These are the mainstream default — Q4_K_M is the workhorse.

I-quants

I-quants are the IQ names: IQ2_XXS, IQ3_XS, IQ4_XS, and friends. They use a more sophisticated, codebook/lookup-style encoding that squeezes noticeably more quality out of very low bit-rates (the 2–3 bit range) than any K-quant can. The trade-off: their dequantization is more complex, so on some hardware — CPUs and older GPUs especially — they can decode a bit slower than a comparable K-quant. If you need to run something that barely fits, I-quants are how you get there without falling off a quality cliff.

imatrix (importance matrix)

An imatrix is not a format — it is a calibration file. It is produced by running sample text through the model and recording which weights are activated most, i.e. which ones are "important". During quantization, that map is used to allocate precision more intelligently.

Key points:

  • It is applied at build time; it does not change the file format, the file size, or how you run the model.
  • It helps most at low bit-rates, and I-quants in particular are usually built with an imatrix.
  • A quant built with a good imatrix generally reads better than the same level built without one — but the calibration data matters, so it is not magic.

Picking in practice