Plain-language explainers for running open-weight models locally.
GGUF, AWQ, GPTQ, EXL2 and MLX — what they are, and what Q4_K_M means.
How model size, quantization and context add up to a memory budget.
Why a longer context costs memory, and what KV-cache quantization trades.
How refusal-direction removal works, and the tradeoffs it introduces. Neutral and factual — not a recommendation for or against.
When a model is writing its answer, it is limited by how fast it can read its weights from memory — not by raw compute. This is the rule of thumb behind ranking hardware by bandwidth.
Mixture-of-experts models activate only a slice of their weights per token, so they decode about as fast as a small model — but you still need memory for the whole thing.
Two different speeds: how fast a model reads your prompt (pp) versus how fast it writes the answer (tg). Long context and RAG bottleneck on the first.
A short decision ladder for choosing a GGUF quant level. Complements the formats explainer and the per-model quant helper tool.
What the K in Q4_K_M means, how I-quants (IQ...) differ, and what an importance matrix (imatrix) actually does during quantization.
CUDA, ROCm, Vulkan and SYCL — how the vendor paths compare for running LLMs locally, and why the fastest backend is now architecture-dependent (RDNA3 vs RDNA4). See the living status board for dated, sourced detail. Reviewed July 2026.
On Apple Silicon, unified memory sets what fits and memory bandwidth sets how fast. Here is how the M-series tiers stack up — including the M3 Pro dip and the fact that there is no M4 Ultra.
Four popular ways to run models locally, and which one fits single-user desktop use versus multi-user serving, by GPU vendor and OS. Opinionated but fair.
A second GPU adds VRAM, not speed. When pooling two cards actually helps, when a single bigger card wins, and the bandwidth/PCIe/power caveats behind the pitch.
You bought the hardware — now what? Realistic local-LLM project ideas grouped by rig, from an 8 GB CPU-only laptop to a multi-GPU box, each with an example catalog model, a runtime, and honest expectations.
What actually runs on an old laptop with no graphics card: which 1–4B models converse acceptably at Q4, the runtime and context settings that make them usable, honest speed expectations, and the things this tier simply cannot do.