NVIDIA, AMD and Intel: the local-LLM software stacks

CUDA, ROCm, Vulkan and SYCL — how mature each vendor path is for running LLMs locally, and what that means for speed and features. Reviewed July 2026.

A GPU's raw specs are only half the story for local LLMs; the software stack decides what actually runs and how fast. Here is the practical picture as of July 2026.

NVIDIA — CUDA

The most mature path by a wide margin. CUDA has the broadest support across runtimes and unlocks the accelerations others lack: FlashAttention, TensorRT-LLM, ExLlamaV2, first-class vLLM, and NVLink for pooling memory across cards. If you want the fewest surprises, this is the default.

AMD — ROCm (and Vulkan)

AMD Radeon cards run today's models, but the stack is less mature and platform matters:

  • ROCm on Linux is the strong path and the one to use for best AMD performance.
  • ROCm on Windows lags. On Windows a Vulkan backend is often the practical choice, and it typically reaches only about 80–90% of the tokens/sec you would get from ROCm on Linux.

Intel — SYCL and Vulkan

Intel Arc also runs models, with two backends. In practice the Vulkan backend is roughly 2× faster than Intel's own SYCL / IPEX-LLM path for llama.cpp-style inference, so prefer Vulkan builds unless a specific workload needs SYCL.

Vulkan — the common denominator

Vulkan runs on all three vendors. It rarely hits the absolute peak of a vendor-native stack, but it is portable and, on AMD/Intel under Windows, often the most reliable option.

What the gap actually costs you

Non-NVIDIA cards have no FlashAttention-3, no TensorRT-LLM, no ExLlamaV2, and no NVLink pooling. Crucially, those mostly affect prompt processing and features, not raw decode: token generation is bandwidth-bound and that roofline holds across vendors (Bandwidth vs compute). So an AMD/Intel card can decode at the speed its bandwidth implies while still trailing on long-prompt ingestion (pp vs tg).

Each AMD and Intel GPU page in the catalog carries a dated ROCm/Vulkan caveat consistent with this — see the GPU pages. This guidance was reviewed July 2026; the ROCm/Vulkan gap narrows over time.