The smallest usable local LLM setup: 8 GB RAM, CPU only
What actually runs on an old laptop with no graphics card: which 1–4B models converse acceptably at Q4, the runtime and context settings that make them usable, honest speed expectations, and the things this tier simply cannot do.
8 min read Reviewed July 2026
On this page
TL;DR
The smallest genuinely usable local setup is a 1–4B model at Q4_K_M, run on the CPU with a memory-mapping runtime like llama.cpp, Ollama, LM Studio or koboldcpp. On a typical 8 GB laptop with no graphics card this holds short conversations, drafts short text and answers simple questions at a readable but not fast pace — think single-digit to low-double-digit tokens per second. It is a real assistant for small tasks; it is not a coding partner, a long-document reader, or an agent.
What does "usable" mean at this tier?
"Usable" here means a model that loads inside 8 GB of system RAM, replies at a speed you can read along with, and stays coherent for a normal back-and-forth chat. It does not mean fast, and it does not mean smart enough for hard work — it means the thing runs and holds a conversation without swapping to disk or falling apart.
Two limits define this tier, and they are both about memory, not compute:
- The model plus everything else must fit in RAM. On an 8 GB machine the OS, your browser and the runtime already claim a few gigabytes, so the model file plus its working memory realistically wants to stay under ~4–5 GB.
- Generation speed is bound by memory bandwidth, not the CPU. A laptop reads its weights from ordinary DDR RAM every token, and that pipe is narrow compared to a graphics card — which is why a small model feels tolerable and a big one feels frozen. See why bandwidth, not raw compute, sets the speed.
Which small models actually converse acceptably?
At Q4, general-purpose chat models in the 1–4B range are the sweet spot: big enough to stay coherent, small enough to fit. From our catalog, these are the realistic conversational picks (sizes are rule-of-thumb estimates — see the note under the table):
| Model | Params (B) | Q4_K_M size, GB (est.) | 8 GB fit | Good at |
|---|---|---|---|---|
| Qwen3-4B | 4.0 | ~2.3 | comfortable | well-rounded general chat, top of the size tier |
| Phi-4-mini-instruct | 3.8 | ~2.2 | comfortable | instruction-following, short reasoning |
| granite-4.0-h-micro | 3.2 | ~1.8 | comfortable | tidy, business-style short answers |
| SmolLM3-3B | 3.0 | ~1.7 | comfortable | lightweight everyday chat |
| Qwen3-1.7B | 2.0 | ~1.2 | roomy | quick replies on very old machines |
| DeepSeek-R1-Distill-Qwen-1.5B | 1.8 | ~1.0 | roomy | short step-by-step reasoning |
| Qwen3-0.6B | 0.75 | ~0.5 | trivial | the floor — simple prompts only |
Params are from our live catalog. Q4_K_M sizes are estimates from the rule of thumb that a Q4_K_M file occupies roughly 0.5–0.6 GB per billion parameters (~4.5 bits per weight); the exact download size is on each model's page. Working RAM in use is the file plus the KV cache and a little overhead on top.
What quant should I download?
Q4_K_M is the starting point at this tier — the standard balance of size and quality. On 8 GB the quant ladder is the same as everywhere else; the only difference is that the ceiling comes down fast, so work top down until it fits:
Choosing a quant on 8 GB, top down
- Does Q6_K or Q5_K_M still leave comfortable headroom? Take the higher quant — cleaner output at no real cost.
- Does Q4_K_M fit at your context? This is the standard default at this tier — start here.
- Only Q3 or Q2 fits? Don't. Drop to a smaller model at Q4_K_M instead — a healthy quant on a smaller model beats a bigger model crushed to 2–3 bits.
How do I actually run it on an old laptop?
Any of the mainstream local runtimes work CPU-only. LM Studio and koboldcpp give you a GUI; Ollama is a simple command-line pull-and-run; llama.cpp is the engine underneath most of them.
Getting a small model running on CPU
- Pick a runtime. LM Studio or koboldcpp for a window you click; Ollama for one command; llama.cpp if you want the raw engine.
- Download a GGUF at Q4_K_M of a 1–4B model from the table above.
- Confirm memory-mapping (mmap) is on. It is the default in llama.cpp-based tools — the model is mapped from disk instead of fully copied into RAM, which is what lets a tight machine load at all.
- Set threads to your physical core count, not more. Four physical cores → four threads. Hyper-threads rarely help CPU inference and can hurt.
- Start the context modest (2K–4K). You can raise it later if RAM allows — see the next section.
Why does context length matter so much here?
The KV cache — the model's memory of the conversation so far — grows with every token of context, and on a CPU setup it comes straight out of the same 8 GB the model already lives in. A model that fits fine at 2K context can push you into swapping at 32K. On this tier, treat a large context as a luxury you spend RAM on, not a free setting.
How fast will it actually be?
Honestly: readable, not snappy. We do not yet have in-house measured CPU-only numbers at this tier — our measured dataset is all discrete GPUs at 7B and up — so treat the following as a rule of thumb, not a benchmark.
Token generation speed is roughly memory bandwidth ÷ bytes read per token. A dual-channel DDR4 laptop moves on the order of ~40–60 GB/s peak, and a 1–4B model at Q4 reads on the order of ~1–2.5 GB per token. Because real CPU inference only reaches a fraction of that peak bandwidth (and drops further the moment the machine starts swapping), the achievable result lands below that raw division — in the single digits to low double digits of tokens per second, a few words a second, fast enough to read along with, slow enough that you would not want it writing pages. Newer laptops with DDR5 or LPDDR5X do better; a machine that has to swap does far worse.
What can this setup NOT do?
This is the honest part. An 8 GB CPU-only setup is a small-tasks assistant, and asking more of it wastes your afternoon:
- Long documents. Big context is expensive here (the KV-cache problem above), so summarizing a long report or a whole codebase is out. Short passages only.
- Serious coding. A 1–4B model can autocomplete a line or explain a snippet, but it will not reliably write, debug or refactor real programs. Use a hosted model for that.
- Agentic / tool-use work. Multi-step planning, calling tools, staying on task across many turns — small models drift and lose the thread. Not a reliable agent.
- Vision, audio and images. These are text chat models. Seeing pictures, transcribing audio or generating images needs different, specialized models (we carry some ASR/OCR/embedding models, but they are not general chat).
- Being a dependable fact source. Smaller models confabulate more. Good for phrasing, brainstorming and simple questions; check anything that matters.
Can I run a 7B or 8B model on 8 GB of RAM?
Usually not comfortably. A 7B at Q4 is ~4–5 GB before context, which leaves almost nothing for the OS, the KV cache and your other apps — so it either swaps to disk and crawls, or won't load. Stay at 1–4B, or add RAM.
Do I need a graphics card at all?
No. Every model in the table above runs on the CPU alone. A GPU makes generation faster and lets you run bigger models, but it is not required to converse with a small model.
Is Ollama, LM Studio, llama.cpp or koboldcpp fastest on CPU?
They share the same underlying engine for CPU inference, so raw speed is similar — pick on interface. What actually moves your tokens/sec is RAM bandwidth, quant and thread count, not which wrapper you chose. See the runtimes guide.
Will more CPU cores make it much faster?
Up to a point. Matching threads to your physical cores helps; past that you hit the memory-bandwidth wall and extra cores do little, because the bottleneck is feeding weights from RAM, not compute.
Sources & data
- Model names and parameter counts: LLM Torrents live catalog (
params_total_b), checked 2026-07. Per-model Q4 download sizes are on each model page. - Q4_K_M size estimates: rule of thumb from the bits-per-weight relationship (~4.5 bits/weight for Q4_K_M), llama.cpp quantization tables (checked 2026-07) — labelled estimates, not benchmarks.
- CPU generation-speed figures: rule-of-thumb from the memory-bandwidth relationship (speed ≈ bandwidth ÷ bytes per token); llama.cpp project documentation and discussions on CPU inference (checked 2026-07). We hold no in-house measured CPU-only tok/s at this tier.