VRAM guide: what fits in 8, 12, 16 and 24 GB
How model size, quantization and context add up to a memory budget.
4 min read Reviewed July 2026
TL;DR
Whether a model runs on your machine comes down to one question: do the weights, the KV cache and the runtime overhead all fit in memory at once? As rule-of-thumb starting points for 4-bit quants at a moderate context: 8 GB runs models up to about 7–8B, 12 GB stretches to 13–14B, 16 GB runs 13–14B comfortably, and 24 GB handles 30–34B. If it doesn't fit: drop the quant, shrink the context, or offload layers to the CPU.
What actually uses the memory?
First, what "memory" means here: on a discrete GPU it's VRAM; on a Mac it's the unified memory shared with the whole system (see which Mac chip for LLMs); on a CPU-only setup it's system RAM. The pool differs, but the budgeting logic is identical — three things consume it.
- The weights. The model file itself. As a rule of thumb, parameters (in billions) × bits-per-weight ÷ 8 ≈ gigabytes — so a 7B model at 4-bit is about 3.5–4 GB of weights, and the same model at 8-bit is about 7–8 GB. The bit-rate is set by quantization.
- The KV cache. The model's working memory, which grows with how much context you use — a few hundred MB for short prompts, far more at long context. The context and KV cache explainer covers why.
- Overhead. The runtime, CUDA/Metal buffers, and headroom. Budget roughly 1–2 GB, and leave some memory free for your display and other apps.
The practical target is to keep all three comfortably under your total memory rather than filling it to the brim. A model that loads with almost nothing to spare is a model that fails the moment you paste in a long document.
Rough fits by memory size
These are rule-of-thumb starting points, not guarantees. They assume 4-bit (Q4-class) GGUF quants and a moderate context window; exact fit depends on the specific quant, your context length, and the architecture — mixture-of-experts models show different behaviour, because all of their parameters must fit even though only a few are active per token.
| Memory | Comfortable fit | Stretch |
|---|---|---|
| 8 GB | Up to ~7–8B at 4-bit | Larger models may load, but leave little room for context |
| 12 GB | 7–8B at higher precision | 13–14B at 4-bit |
| 16 GB | 13–14B at 4-bit | Some 20B-class models at 4-bit with a modest context |
| 24 GB | 30–34B at 4-bit, or mid-size models at higher precision | 30–34B at 4-bit with long context — the KV cache becomes the pressure point |
Which quant to pair with the space you have is its own decision — which quant should I pick? walks the ladder.
What if it doesn't fit?
When it doesn't fit, in this order
- Drop the bit-rate. Moving from Q8 to Q4 roughly halves the weight size — usually the biggest single saving available.
- Reduce context. A smaller context window shrinks the KV cache; if you don't need long documents, this costs you nothing.
- Quantize the KV cache. The cache itself can be stored at 8-bit instead of 16-bit, roughly halving its footprint — see context and the KV cache.
- Offload. GGUF runtimes can split a model across GPU and CPU, running the layers that fit on the GPU and the rest on the CPU. The model runs when it wouldn't otherwise fit in VRAM — at reduced speed, since the CPU-side layers stream from slower system memory (see bandwidth vs compute).
If you've reached the bottom of that list and it still doesn't fit, the honest answer is a smaller model — a healthier quant of a smaller model generally beats a big model crushed to the lowest bit-rates.
FAQ
Does system RAM add to my VRAM?
Not directly — a GPU only computes against weights held in its own VRAM. What offloading does is run some layers on the CPU from system RAM instead, so the whole model works, but those layers run at system-RAM speed and drag generation down. RAM buys fit, not free capacity.
Do mixture-of-experts models fit differently?
For memory, no: all parameters must fit, even though only a few experts are active per token. The small active-parameter count helps speed, not fit. See MoE vs dense.