LLM Torrents

VRAM guide: what fits in 8, 12, 16 and 24 GB

How model size, quantization and context add up to a memory budget.

Whether a model runs on your machine comes down to one question: does everything fit in memory at once? On a GPU that means VRAM; on a Mac it means the unified memory shared with the system; on a CPU-only setup it means system RAM. Three things consume that budget.

What uses memory

  1. The weights. This is the model file itself. A rough estimate: parameters (in billions) × bits-per-weight ÷ 8 = gigabytes. A 7B model at 4-bit is about 3.5–4 GB of weights; the same model at 8-bit is about 7–8 GB.
  2. The KV cache. Working memory that grows with how much context you use. A few hundred MB for short prompts, but it scales up with long contexts (see the context & KV-cache explainer).
  3. Overhead. The runtime, CUDA/Metal buffers, and headroom. Budget roughly 1–2 GB, and leave some VRAM free for your display and other apps.

A practical target is to keep the weights + KV cache + overhead comfortably under your total memory, rather than filling it to the brim.

Rough fits by memory size

These are general guidelines for 4-bit (Q4-class) GGUF quants with a moderate context window. Exact fit depends on the quant, context length, and architecture (mixture-of-experts models behave differently), so treat them as starting points, not guarantees.

  • 8 GB — comfortably runs models up to about 7–8B at 4-bit. Larger models may load but leave little room for context.
  • 12 GB — headroom for 7–8B at higher precision, or 13–14B at 4-bit.
  • 16 GB — 13–14B comfortably; some 20B-class models at 4-bit with a modest context.
  • 24 GB — 30–34B at 4-bit, or mid-size models at higher precision with long context.

When it doesn't fit

  • Drop the bit-rate. Moving from Q8 to Q4 roughly halves the weight size.
  • Reduce context. A smaller context window shrinks the KV cache.
  • Offload. GGUF runtimes can split a model across GPU and CPU, running the layers that fit on the GPU and the rest on the CPU. This lets a model run when it wouldn't otherwise fit entirely in VRAM.

To check a specific model against your hardware, use the calculator, which estimates the memory budget for a given quant and context.