Which quant should I pick?

Pick your GPU and a model, and we'll rank its quants from highest to lowest quality — with the fit verdict, size, an estimated speed, and a plain-English note on what you give up at each step down. No account needed.

1 · Your GPU

  • No matching GPU in the list — you can type the model manually.

Search for your card above (Apple Silicon routes to unified memory automatically). Without a GPU we'll still show sizes and the quality guidance below.

Used to judge partial CPU offload when a quant overflows VRAM.

Save your rig for one-click answers.

Your rig lives in this browser. A free account keeps it across the calculator, model pages and this helper.

Save my rig

2 · Model & context

Lower precision shrinks the KV cache (fits more context).

The quant ladder, top to bottom

Highest quality first. Each step down trades quality for a smaller file. Pick a model above to see which rungs actually fit your GPU, their sizes and estimated speeds.

  • F16 Full 16-bit weights: no quantization loss at all, but roughly twice the size of an 8-bit quant and rarely worth it over Q8 for local use.
  • Q8_0 Effectively lossless in community experience — the closest you get to the original model, at the largest practical GGUF size.
  • Q6_K Very close to Q8 in everyday use; the usual choice when Q8 is just too big to fit your memory.
  • Q5_K_M High quality with a noticeably smaller footprint than Q6 — a popular 'quality first, but it still fits' pick.
  • Q5_K_S A little smaller than Q5_K_M for a small quality trade; useful when you need to claw back some memory.
  • Q4_K_M The community default: minor quality loss versus Q5/Q6, usually unnoticeable outside edge tasks, at a much smaller size.
  • Q4_K_S Slightly smaller than Q4_K_M for a modestly larger quality trade; handy when Q4_K_M just misses your VRAM.
  • IQ4_XS An i-quant that packs 4-bit weights tighter than Q4_K_S — similar quality in community reports and a smaller file, though slightly heavier to decode on some runtimes.
  • Q3_K_M Noticeably lighter than Q4 with a real, more visible quality drop — a squeeze-it-in option rather than a default.
  • Q3_K_S Smaller and rougher than Q3_K_M; generally a last resort before dropping to Q2.
  • Q2_K The smallest common quant — expect clearly degraded output. Community advice is usually to run a smaller model at a higher quant before choosing Q2.

Guidance reflects community experience, not measured benchmarks — quality loss is usually more noticeable on smaller models than large ones.