On a 16 GB GPU, 5 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-Embedding-8B at Q4_K_M (needs ~6.4 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 16 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
RAG is a pipeline, not a single model — and 16 GB fits the whole retrieval side of it comfortably. The models on this page are the catalog's RAG components (embedders for indexing and search, rerankers for sharpening results); they're small enough that most leave room to run a chat model alongside them. For the generator half of the pipeline, see what 16 GB runs.
Qwen3-Embedding-8B at Q4_K_M · 7.6B params · needs ~6.4 GiB
One command to run it (llama.cpp):
llama-server -m Qwen3-Embedding-8B-Q4_K_M.gguf -c 8192 -ngl 999 --embeddings| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-Embedding-8B Qwen | Q4_K_M Runs fully on GPU | ~6.4 GiB |
| Qwen3-Reranker-4B Qwen | BF16 Runs fully on GPU | ~9.9 GiB |
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
| Qwen3-Embedding-0.6B Qwen | Q8_0 Runs fully on GPU | ~2 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
Yes — chat models don't produce the document vectors retrieval needs. A dedicated embedding model indexes your documents and embeds each query; it's a small, fast component compared with the generator, and it only needs to run at indexing time and per query.
A reranker re-scores the passages your vector search returns, so the most relevant ones actually reach the generator. It's optional, but often one of the cheapest quality upgrades in a RAG stack — reranker models are small and only score a handful of candidates per query.
Usually, yes. Embedders and rerankers are small next to a generator, and many stacks load the embedder only for indexing and queries rather than keeping it resident. The practical approach: budget the card for your generator first, then confirm the combination in the fit calculator.
Any instruct model from the general 16 GB list that leaves room for your retrieval components and — importantly — for context, since RAG stuffs retrieved passages into the prompt. A mid-size model with context headroom often beats the largest model that technically fits.
Yes — every retrieved passage lands in the prompt, so RAG workloads run long contexts as a matter of course, and the KV cache grows to match. When sizing your generator, check the fit calculator at the context you'll really use with retrieval, not the default.
selected to compare · pick at least 2