Which quant should I pick?

You have picked a model and it ships in a dozen quant levels. Which one? This is a quick ladder. For what the formats and the Q4_K_M-style names actually mean, start with Quantization formats explained; this page assumes GGUF and focuses only on which level to grab.

The one rule

Fit first, then take the highest bit-rate that still fits.

Lower bits = smaller file, less memory, and (because fewer bytes are read per token) slightly faster decode — at the cost of drifting further from the original model. Higher bits = closer to the original, larger, slightly slower.

The ladder

1. Start at Q4_K_M. The common default: a strong balance of size and quality for most models.
2. Have memory to spare? Step up. Q5_K_M, then Q6_K, then Q8_0 (near-lossless, but roughly twice the size of a 4-bit quant). Each step keeps more of the original behaviour.
3. Tight on memory? Step down carefully. Q4_K_S shaves a little more; below 4-bit, look at the I-quants (IQ3, IQ2) which hold up better than old low-bit quants at 2–3 bits (K-quants, I-quants and imatrix).
4. Mind the model size. Low bit-rates hurt small models more than large ones — a 3-bit 70B often reads better than a 3-bit 7B. Give small models more bits when you can.

Don't eyeball it — use the tool

The catalog has a quant helper: pick your GPU and a model and it ranks that model's actual quants by quality and fit, and shows the cost of each step down. This ladder is the intuition; the helper is the hardware-specific answer. Cross-check your budget against the VRAM guide, and remember a bigger quant also means slightly slower decode (Bandwidth vs compute).