On a 6 GB GPU, 3 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-Embedding-4B at Q4_K_M (needs ~4.2 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 6 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
RAG is a pipeline, not a single model — and even 6 GB covers the retrieval side of it. The models on this page are the catalog's RAG components (embedders for indexing and search, rerankers for sharpening results); they're small, but on a card this size they earn their keep by taking turns rather than staying loaded. For the generator half of the pipeline — the part that actually writes answers — see what 6 GB runs.
Qwen3-Embedding-4B at Q4_K_M · 4B params · needs ~4.2 GiB
One command to run it (llama.cpp):
llama-server -m Qwen3-Embedding-4B-Q4_K_M.gguf -c 8192 -ngl 999 --embeddings| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
| Qwen3-Embedding-0.6B Qwen | Q8_0 Runs fully on GPU | ~2 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
Yes — because the pieces don't all need to be loaded at once. The embedder runs at indexing time and briefly per query, the reranker (if you use one) only scores a handful of passages, and the generator — the one piece that stays resident — gets the card to itself. Think of it as a pipeline in time slices, not one giant model.
No. Most local RAG stacks load the embedder to build the index, unload it, and run the generator afterwards. For query-time embedding, small embedders are quick enough on the CPU that many setups never put them back on the GPU at all.
Often, yes. A reranker re-scores the top search hits so the best passages reach the generator, and it only handles a handful of candidates per query — so the small reranker class adds little memory and can run on the CPU if the card is spoken for. On a tight card it's one of the few quality upgrades that barely costs generator headroom.
A small instruct model from the general 6 GB list — and leave context headroom, because retrieved passages land in the prompt and the KV cache grows to match. At this tier a smaller model with room to breathe usually beats the largest model that technically fits.
Because those are the retrieval-specific parts of a RAG pipeline — the components the catalog tags for RAG. The generator is an ordinary instruct model, so it lives on the general 6 GB fit list rather than here. That's also why the headline can crown an embedding model as most capable: it's the most capable retrieval component, not the model that writes your answers.
selected to compare · pick at least 2