On a 48 GB GPU, 6 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-Reranker-8B at BF16 (needs ~18.4 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 48 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
RAG is a pipeline, not a single model — and 48 GB runs the whole pipeline as a resident, always-on service. The models on this page are the catalog's retrieval components (embedders for indexing and search, rerankers for sharpening results); all of them fit here with room to spare, so the real question is how much card to leave the generator — the piece that actually writes answers. Choose it from what 48 GB runs.
At 48 GB the question stops being fit and becomes service design. Keep an embedder resident so queries embed instantly, keep a reranker warm, and the retrieval side still leaves the lion's share of the card to the generator. The trade-off worth actually thinking about: a larger generator class, or a mid-size generator with very long context for more retrieved passages per answer. For retrieval-heavy work, context often pays better than raw size.
Qwen3-Reranker-8B at BF16 · 8.2B params · needs ~18.4 GiB
One command to run it (llama.cpp):
vllm serve Qwen/Qwen3-Reranker-8B --max-model-len 8192| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-Reranker-8B Qwen | BF16 Runs fully on GPU | ~18.4 GiB |
| Qwen3-Embedding-8B Qwen | Q4_K_M Runs fully on GPU | ~6.4 GiB |
| Qwen3-Reranker-4B Qwen | BF16 Runs fully on GPU | ~9.9 GiB |
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
| Qwen3-Embedding-0.6B Qwen | Q8_0 Runs fully on GPU | ~2 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
For the retrieval side, comfortably — embedders and rerankers run on far smaller cards, as the lower tiers of these pages show. What 48 GB is actually for: a larger generator class, the whole pipeline resident at once, and the long contexts that let retrieval put more passages in front of the model per answer.
Yes — a resident embedder, a resident reranker, and a generator with generous context is a realistic single-card setup at this tier. The caveat is concurrency: each parallel request carries its own KV cache, so serving several users multiplies the context cost. Size for your real request pattern, not a single chat.
For retrieval-heavy work, context is usually the better spend — more retrieved passages in the prompt tends to improve grounded answers more than the next step up in model size does. But it's workload-dependent: complex reasoning over few documents favors the bigger model. Try both; the fit calculator shows what each costs.
No — the retrieval components don't scale with your card, which is why this list looks like the smaller tiers'. Pick them for your corpus and language needs, not for the GPU. The card's size shows up in what you pair them with, not in the retrieval side itself.
The larger instruct classes from the general 48 GB list — sized, as always with RAG, at the context you'll really use, since retrieved passages inflate every prompt and the KV cache grows faster on big models. A near-lossless quant of a mid-size model with huge context is a legitimate alternative to the largest class that fits.
selected to compare · pick at least 2