Which quant should I pick?

A short decision ladder for choosing a GGUF quant level. Complements the formats explainer and the per-model quant helper tool.

4 min read Reviewed July 2026

TL;DR

Fit first, then take the highest bit-rate that still fits. Start at Q4_K_M, the standard default; step up to Q5/Q6/Q8 if you have memory to spare, and step down carefully if you don't. Below 4-bit, reach for an I-quant — or reconsider and run a smaller model at a healthier quant instead.

You have picked a model and it ships in a dozen quant levels. Which one? This page is a quick decision ladder. For what the formats and the Q4_K_M-style names actually mean, start with Quantization formats explained; this page assumes GGUF and focuses only on which level to grab.

What are you trading?

Every step down the ladder trades fidelity for footprint:

  • Lower bits — smaller file, less memory, and (because fewer bytes are read per token) slightly faster decode, at the cost of drifting further from the original model.
  • Higher bits — closer to the original behaviour, larger, slightly slower (Bandwidth vs compute).

And remember the weights are not your whole memory budget: context eats memory too (Context and the KV cache), so judge "fits" at the context length you actually intend to run.

The decision ladder

Choosing a quant, top down

  1. Start at Q4_K_M. The common default: a strong balance of size and quality for most models. If you check nothing else, check whether this fits.
  2. Have memory to spare? Step up. Q5_K_M, then Q6_K, then Q8_0 (near-lossless, but roughly twice the size of a 4-bit quant). Each step keeps more of the original behaviour.
  3. Tight on memory? Step down carefully. Q4_K_S shaves a little more; below 4-bit, look at the I-quants (IQ3, IQ2), which hold up better than the old low-bit quants at 2–3 bits (K-quants, I-quants and imatrix).
  4. Only 2–3-bit fits? Reconsider the model. A smaller model at Q4/Q5 occupies a similar footprint and often reads better than a bigger model crushed to 2–3 bits — especially once the small-model rule below is in play.

Does model size change the answer?

Yes — and it is the most common way people go wrong at the bottom of the ladder. Quantization loss is not spread evenly across model sizes: large models absorb low bit-rates far more gracefully than small ones. A 3-bit 70B often reads better than a 3-bit 7B.

Bigger model at Q2, or smaller model at Q4?

Usually the smaller model at Q4. The 2-bit range is where quality falls away fastest, and a healthier quant of a smaller model tends to give more coherent output for a similar footprint. Large models tolerate low bits better than small ones, so the gap narrows at 70B-class sizes — but test on your task.

Does a lower quant actually run faster?

Slightly, in token generation — fewer bytes are read from memory per token, and local inference is usually bandwidth-bound (Bandwidth vs compute). Treat it as a bonus, not a reason to pick a lower quant than you need.

Do these levels apply to AWQ, GPTQ or EXL2?

This ladder is written for GGUF, where one model ships at many levels. AWQ and GPTQ are mostly 4-bit schemes, and EXL2 mixes bit-rates within one file — see Quantization formats explained for how the formats differ.

Don't eyeball it — use the tool

This ladder is the intuition; the quant helper is the hardware-specific answer. Cross-check your memory budget against the VRAM guide at your real context length, and remember a bigger quant also means slightly slower decode.