On a 24 GB GPU, 6 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-Reranker-8B at BF16 (needs ~18.4 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 24 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
RAG is a pipeline, not a single model — and at 24 GB the retrieval side is the easy part. Everything on this page is a retrieval component (embedders for indexing and search, rerankers for sharpening results); the real decision at this tier is how much of the card to hand the generator, because 24 GB puts the 30B class in reach. Pick that half from what 24 GB runs and treat the models below as the supporting cast.
Qwen3-Reranker-8B at BF16 · 8.2B params · needs ~18.4 GiB
One command to run it (llama.cpp):
vllm serve Qwen/Qwen3-Reranker-8B --max-model-len 8192| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-Reranker-8B Qwen | BF16 Runs fully on GPU | ~18.4 GiB |
| Qwen3-Embedding-8B Qwen | Q4_K_M Runs fully on GPU | ~6.4 GiB |
| Qwen3-Reranker-4B Qwen | BF16 Runs fully on GPU | ~9.9 GiB |
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
| Qwen3-Embedding-0.6B Qwen | Q8_0 Runs fully on GPU | ~2 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
Because this page ranks retrieval components, and the larger rerankers are listed at full precision while embedders run quantized — so a reranker tops the table. It's the most capable retrieval piece, not the model that writes your answers; that's the generator, which you choose from the general 24 GB list.
Yes — that's the natural 24 GB setup. Keep the generator resident, run the embedder on demand (indexing is a batch job, and query embedding is cheap), and slot a small reranker into what's left. Verify the exact combination in the fit calculator at the context you'll really use, since retrieved passages inflate the prompt.
The small ones can; the full-precision heavyweights shouldn't, because they compete directly with the generator's memory and context headroom. As a rule of thumb: generator and its context first, small retrieval components resident if convenient, big rerankers loaded per pass.
It shifts the trade-off toward context. Retrieved passages fill the prompt and the KV cache grows to match — faster on big models — so a generator that barely fits leaves no room for retrieval to breathe. A model one notch smaller, run with generous context, often answers better from retrieved documents than the largest fit.
No — the retrieval side runs on far smaller cards, as the lower tiers of these pages show. What 24 GB buys is the generator: the 30B class fully on the GPU, with enough margin to run retrieval alongside it instead of juggling loads.
selected to compare · pick at least 2