Multi-GPU for local LLMs: when a second card is worth it
A second GPU adds VRAM, not speed. When pooling two cards actually helps, when a single bigger card wins, and the bandwidth/PCIe/power caveats behind the pitch.
5 min read Reviewed July 2026
On this page
TL;DR
A second GPU adds VRAM, not speed. Two cards let you load a bigger model by splitting its weights across both, but token generation is memory-bandwidth-bound, so a layer-split model decodes at roughly one card's speed — not double. Add a second card when a model you want won't fit on one; don't add one expecting faster tokens/sec. Matched cards on a fast interconnect can run tensor-parallel and use both at once, but that's a server setup, not the desktop default.
What does a second GPU actually buy you?
The one thing a second card reliably adds is memory. Two 16 GB cards give a 32 GB pool to hold weights and the KV cache. That's the whole pitch: models too big to fit on one card can fit across two.
What it does not buy, in the common desktop setup, is proportionally faster generation. Token generation is bound by memory bandwidth, and splitting a model across two cards doesn't raise the bandwidth any single token sees.
In short:
- A second card adds: pooled VRAM — bigger models and higher-fidelity quants can fit.
- It does not add: per-token memory bandwidth — single-stream generation speed stays at roughly one card's pace.
Does VRAM add up? Does bandwidth?
VRAM adds; bandwidth does not — the single most important thing to grasp before buying.
| Resource | Two 16 GB cards | Behaves like |
|---|---|---|
| Total VRAM | 32 GB | one 32 GB pool (weights split across cards) |
| Bandwidth per token | each card's own | a single card, not the sum |
| Prompt-processing compute | both cards | can overlap; helps pp more than tg |
So a pair of smaller cards and one big card of the same total VRAM are not equivalent: the single big card keeps its full bandwidth for every token, while the pair pays a coordination cost and, under a layer split, decodes at about one card's speed.
Layer split or tensor parallel?
There are two ways to use two cards, and they behave very differently.
Layer split (llama.cpp default)
- Each card holds some layers
- Cards run in sequence — one busy at a time
- Tolerates mismatched cards
- Decode speed ≈ one card
Tensor parallel (vLLM etc.)
- Each card holds part of every layer
- Cards run in parallel — both busy
- Wants matched cards + fast link
- Can raise throughput, esp. batched serving
Most desktop tools (Ollama, LM Studio, llama.cpp) do a layer split by default: simple, robust, and happy to mix two different cards — but only one GPU works at a time, so you get the memory of two cards at roughly the speed of one. Tensor-parallel serving (vLLM) genuinely uses both cards at once, but wants identical cards and a fast interconnect, and it targets multi-user serving rather than a single chat.
What about the link between the cards?
Cards talk to each other over PCIe — or NVLink on the few consumer/workstation cards that support it. PCIe is far slower than the memory on a card: a PCIe 4.0 x16 link moves on the order of 32 GB/s, while a modern card's on-board memory runs in the hundreds of GB/s to over 1 TB/s.
Under a layer split, that link is crossed only about once per token (handing off between layer groups), so the penalty is small. Under tensor parallelism it's crossed constantly, which is why a fast interconnect matters there. And if the two cards sit on different CPU sockets, NUMA effects pile latency on top.
When is a second card worth it?
Decide, top down
- Does the model you want already fit on one card? Then a second card won't speed it up — spend the money elsewhere.
- Does it fit on one card only at a painful quant? A second card can let you run a higher-fidelity quant of the same model.
- Is the model simply too big for one card? This is the real case for a second GPU — pooled VRAM lets it load at all.
- Do you need faster tokens/sec, not a bigger model? Wrong tool — a single higher-bandwidth card beats two slower ones for decode.
- Are you serving many concurrent users? Then tensor-parallel across matched cards can raise total throughput — a server decision, not a desktop one.
Will two GPUs double my tokens per second?
No. Under the common layer-split setup, decode runs at roughly one card's speed — the second card adds VRAM, not decode bandwidth. Only tensor-parallel serving with matched cards uses both at once.
Do my two cards have to be identical?
For a layer split, no — llama.cpp-style tools happily mix, say, a 24 GB and a 12 GB card, pooling to 36 GB. For tensor-parallel serving, matched cards and a fast link matter a lot.
Is one big card better than two smaller ones?
For the same total VRAM and decode speed, a single big card is simpler and keeps full bandwidth per token. Two cards win when they're cheaper per GB of VRAM, or when no single card has enough memory.
Does NVLink make a second card worth it?
NVLink speeds up the inter-card link, which mainly helps tensor-parallel workloads. For a desktop layer split it changes little, because the link is barely used per token.
Sources & data
- PCIe 4.0 x16 throughput (~32 GB/s aggregate): PCI-SIG PCI Express Base Specification (checked 2026-07). Per-card on-board memory bandwidth figures are manufacturer / TechPowerUp specs — see each GPU page.
- Multi-GPU behaviour (layer split vs tensor parallel): llama.cpp and vLLM project documentation (checked 2026-07).