On a 12 GB GPU, 5 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-Embedding-8B at Q4_K_M (needs ~6.4 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 12 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
RAG is a pipeline, not a single model — and at 12 GB the retrieval side stops asking for compromises. The models on this page are the catalog's RAG components (embedders for indexing and search, rerankers for sharpening results); this is the tier where mid-size rerankers show up at full precision, while everything here stays small next to the generator. For that half of the pipeline, see what 12 GB runs.
A reranker re-scores the passages your vector search returns, so the most relevant ones actually reach the generator — often the cheapest quality upgrade in a RAG stack, because it only touches a handful of candidates per query. At 12 GB you can run one on the GPU without denting the generator's budget much, or load it on demand alongside the embedder and keep the card clear the rest of the time.
Qwen3-Embedding-8B at Q4_K_M · 7.6B params · needs ~6.4 GiB
One command to run it (llama.cpp):
llama-server -m Qwen3-Embedding-8B-Q4_K_M.gguf -c 8192 -ngl 999 --embeddings| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-Embedding-8B Qwen | Q4_K_M Runs fully on GPU | ~6.4 GiB |
| Qwen3-Reranker-4B Qwen | BF16 Runs fully on GPU | ~9.9 GiB |
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
| Qwen3-Embedding-0.6B Qwen | Q8_0 Runs fully on GPU | ~2 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
Mostly comfort. The retrieval components that run on smaller cards run here with room to spare, mid-size rerankers fit at full precision, and — more importantly — the general 12 GB list gives you stronger generators to pair them with. The retrieval side was never the bottleneck; the generator options are what grow with the card.
Usually, yes. It re-scores only the top search hits per query, so it works in short bursts rather than sitting on memory the way a generator does — and sharper passages going into the prompt tends to move answer quality more than most other single changes. If your corpus is small or your search results are already clean, it's the first thing to drop.
The smaller embedder classes, comfortably. The larger ones fit too, but they compete with the generator's context headroom — and since embedders only work at indexing time and per query, most stacks load them on demand instead. Budget the generator first, then confirm the combination in the fit calculator.
Any instruct model from the general 12 GB list that leaves real context headroom — retrieved passages fill the prompt, and the KV cache grows to match. A mid-size model with room for long prompts usually serves RAG better than the largest model that technically fits.
selected to compare · pick at least 2