LLM Torrents

Quantization formats explained

GGUF, AWQ, GPTQ, EXL2 and MLX — what they are, and what Q4_K_M means.

Quantization shrinks a model by storing its weights at lower numerical precision. A model trained in 16-bit (FP16/BF16) can be re-packed to roughly 8, 5, 4, or even 3 bits per weight. Fewer bits means a smaller file and less memory to load it, at the cost of some accuracy. The formats below differ mainly in how they pack those bits and which runtimes read them.

The common formats

  • GGUF — the format used by llama.cpp and its ecosystem (LM Studio, Ollama, and many desktop apps). GGUF files are self-contained: weights, tokenizer, and metadata live in one file. It runs on CPU, GPU, or a split of both, which makes it the most portable choice for local use.
  • AWQ (Activation-aware Weight Quantization) — a 4-bit scheme that protects the weights most sensitive to error. Common with GPU runtimes such as vLLM.
  • GPTQ — a widely used 4-bit (sometimes 3- or 8-bit) post-training method, also common on GPU runtimes.
  • EXL2 — the format used by ExLlamaV2. It supports mixed bit-rates within one model and is tuned for NVIDIA GPUs.
  • MLX — Apple's array framework format, built for Apple Silicon (M-series) machines using the unified-memory architecture.

What "Q4_K_M" means

GGUF quant names look cryptic but follow a pattern:

  • Q4 — roughly 4 bits per weight (the headline size/precision trade).
  • _K — a "K-quant", a newer scheme that allocates bits more cleverly across blocks of weights than the older "legacy" quants.
  • _M — the size class within that scheme: S (small), M (medium), L (large). A larger class spends a few more bits on the most important tensors.

So Q4_K_M is a 4-bit K-quant, medium size — a common default that balances file size against how close the output stays to the full-precision model. Q5_K_M and Q6_K keep more precision at a larger size; Q8_0 is near-lossless but roughly twice the size of a 4-bit quant.

Choosing one

Pick the format your runtime supports first, then the bit-rate your memory allows:

  • Running on CPU, a Mac, or a mix of CPU+GPU → GGUF.
  • Running fully on an NVIDIA GPU with vLLM/ExLlama → AWQ, GPTQ, or EXL2.
  • On Apple Silicon and using MLX tooling → MLX.

As a rule of thumb, lower bit-rates diverge more from the original model, and the difference is usually more noticeable on smaller models than on large ones. If a quant fits comfortably, a higher bit-rate (Q5/Q6/Q8) preserves more of the original behaviour; if memory is tight, a 4-bit quant is the usual starting point.