On a 8 GB GPU, 4 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-Embedding-8B at Q4_K_M (needs ~6.4 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 8 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
RAG is a pipeline, not a single model — and 8 GB runs the retrieval side of it without drama. The models on this page are the catalog's RAG components (embedders for indexing and search, rerankers for sharpening results); they're small next to a chat model, and most don't need to stay loaded — which is how a mainstream card ends up running a whole pipeline. For the generator half, see what 8 GB runs.
Usually not. Indexing is a batch job — the embedder does its heavy lifting once, before you ever chat. At query time it embeds one short input, which the smaller embedder classes handle in almost no memory, or on the CPU entirely. The larger embedder class fits this card at sweet-spot quants but takes a real slice of it, so the practical 8 GB pattern is: generator resident, retrieval on demand.
Qwen3-Embedding-8B at Q4_K_M · 7.6B params · needs ~6.4 GiB
One command to run it (llama.cpp):
llama-server -m Qwen3-Embedding-8B-Q4_K_M.gguf -c 8192 -ngl 999 --embeddings| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-Embedding-8B Qwen | Q4_K_M Runs fully on GPU | ~6.4 GiB |
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
| Qwen3-Embedding-0.6B Qwen | Q8_0 Runs fully on GPU | ~2 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
Yes. The retrieval components are small and mostly run on demand, so the card's real job is the generator — the same small instruct class you'd run for regular chat at this tier. The difference RAG makes is context: retrieved passages fill the prompt, so leave the generator room for a longer context than plain chat would need.
The smallest embedder class can sit alongside a compact generator, yes. The larger embedder class is better run on demand — load it to index or embed a query, then give the memory back. Budget the generator first and confirm the combination in the fit calculator at your real context.
Less than most people expect in weights — embedders and rerankers are small, and they don't have to stay resident. The real added cost is context: every retrieved passage lands in the prompt, and the KV cache grows with it. As a rule of thumb, budget for longer prompts rather than for extra models.
Not usually. Larger embedder classes tend to retrieve somewhat better, but chunking your documents well and adding a reranker often move quality more than stepping up the embedder does. On an 8 GB card, a smaller embedder plus a reranker is a very respectable retrieval stack.
Any instruct model from the general 8 GB list that leaves room for context — RAG stuffs retrieved passages into every prompt, so check the fit calculator at the context you'll actually use with retrieval, not the default.
selected to compare · pick at least 2