On a 32 GB GPU, 6 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-Reranker-8B at BF16 (needs ~18.4 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 32 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
RAG is a pipeline, not a single model — and 32 GB is where the pipeline can simply stay loaded. The models on this page are the catalog's retrieval components (embedders for indexing and search, rerankers for sharpening results); at this tier a sensible set of them can sit resident beside a 30B-class generator instead of taking turns. For the generator half, see what 32 GB runs.
Not bigger retrieval models — the same embedders and rerankers appear on smaller tiers, because the retrieval side doesn't grow with your card. What 32 GB buys is the option to stop juggling: an embedder that stays warm for instant query embedding, a reranker that never unloads, and a 30B-class generator with context to spare, all resident at once. Or spend the whole margin on the generator instead — a higher quant of the same class, or the long contexts that retrieval-heavy work eats.
Qwen3-Reranker-8B at BF16 · 8.2B params · needs ~18.4 GiB
One command to run it (llama.cpp):
vllm serve Qwen/Qwen3-Reranker-8B --max-model-len 8192| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-Reranker-8B Qwen | BF16 Runs fully on GPU | ~18.4 GiB |
| Qwen3-Embedding-8B Qwen | Q4_K_M Runs fully on GPU | ~6.4 GiB |
| Qwen3-Reranker-4B Qwen | BF16 Runs fully on GPU | ~9.9 GiB |
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
| Qwen3-Embedding-0.6B Qwen | Q8_0 Runs fully on GPU | ~2 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
Margin, mostly. At 24 GB a 30B-class generator wants nearly the whole card, so retrieval runs on demand; at 32 GB you can keep an embedder and a reranker resident beside that same generator, or spend the difference on longer context or a higher quant. The retrieval components themselves don't change — the room around them does.
A sensible pipeline, yes: a small-to-mid embedder, a small reranker, and a 30B-class generator coexist with room left for context. The full-precision heavyweight rerankers still take a serious bite and compete with the generator's headroom — if you want one of those resident too, check the exact combination in the fit calculator first.
The generator and its context, as a rule of thumb. Retrieval quality gains from bigger components flatten quickly, while RAG's appetite for context — every retrieved passage lands in the prompt — grows with everything you throw at it. Upgrade the generator's quant or context budget before upsizing the retrieval side.
Somewhat, but usually less than people hope. Chunking your documents well and adding a reranker tend to move retrieval quality more than stepping up the embedder class does. With a card this size, the honest upgrade path is better pipeline hygiene, not heavier components.
Any instruct model from the general 32 GB list that leaves real context headroom — retrieved passages inflate every prompt, and the KV cache grows to match. Size the generator in the fit calculator at the context you'll actually use with retrieval before committing the download.
selected to compare · pick at least 2