deepseek-ai

DeepSeek-V3-0324

684.5B parameters · mixture-of-experts · Base · DeepSeek family

Mixture-of-experts — what it means for memory & speed → Long context — how it uses memory →
mit hash verified source matched revision pinned

Which version should I download?

Default pick: GGUF · UD-Q2_K_XL — Unsloth UD-Q2_K_XL dynamic 2-bit quant of DeepSeek-V3-0324 (684.531B total, 256-expert MoE) — recommended balanced pick. MIT.

Set your rig to confirm it fits and to see if a higher-quality quant runs fully on your hardware.

Set your rig

Quantizations

Why these quants? Which quant should I pick? What do these mean?

GGUF · UD-Q2_K_XL

230.59 GB

Set your rig
Est. speed
—
Swarm
1S / 0L webseed OK

Unsloth UD-Q2_K_XL dynamic 2-bit quant of DeepSeek-V3-0324 (684.531B total, 256-expert MoE) — recommended balanced pick. MIT.

Magnet .torrent

Want a different quant? Request it on the board →

Runs fully on

Apple M3 Ultra (512 GB) · UD-Q2_K_XL

Green = the model's best quant fits fully in GPU/unified memory at 8K context. Tap a card for its full "what runs on it" page.

Context

Advertised 160k · usable ≈ 128k (reported — model card)

How we know

Evidence grade: reported — a maintainer claim with no independent evaluation.

Card states structure identical to DeepSeek-V3; the V3 technical-report NIAH figure (clean to 128K) and LongBench v2 48.7 were run on V3, NOT re-run on 0324 — hence reported, not measured.

Reviewed on Jul 18, 2026.

Capabilities (as declared by the maintainer): tool calling

Run it

Runtime completeness (UD-Q2_K_XL torrent): llama.cpp ✅ Ollama – vision sidecar –

  • – Ollama — runs the GGUF directly; no Modelfile bundled
  • – vision sidecar — not a vision model

Context / KV 8,192 tokens · FP16 set above the quant table

Generic commands (no rig set). GPU-offload values assume the model fits on your GPU — set your rig for values tuned to your hardware.

Start the server 
llama-server -m DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf -c 8192 -ngl 999

Use llama-cli in place of llama-server for a one-shot prompt. This model compresses its KV cache, but only on llama.cpp b5137 (April 2025) or newer AND a GGUF converted after that date. Older files -- including some still-popular DeepSeek-R1 and V3-0324 quants -- fall back to the uncompressed layout and use about 5 MB per token instead of 70 KB. Check the quant repo date if long context is the point.

Save as Modelfile next to the GGUF (Modelfile) 
FROM ./DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf
PARAMETER num_ctx 8192
PARAMETER num_gpu 999
PARAMETER stop "<｜end▁of▁sentence｜>"
# Chat template: Ollama uses the template embedded in the GGUF (no TEMPLATE directive needed).
Create 
ollama create deepseek-ai-deepseek-v3-0324 -f Modelfile
Run 
ollama run deepseek-ai-deepseek-v3-0324

GGUF is not a first-class vLLM format.

vLLM GGUF support is experimental and single-file only; prefer safetensors/GPTQ/AWQ for production. If you must, pass the .gguf path to `vllm serve` with --load-format gguf on a recent vLLM.

transformers does not load GGUF weights.

GGUF weights run under llama.cpp, Ollama or LM Studio.

Load DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf, set the context length to 8192 tokens. Set GPU offload to Max (all layers).

MLX runs MLX-format weights only (Apple Silicon). This quant is a GGUF build.

GGUF weights run under llama.cpp, Ollama or LM Studio.

Technical details

Chat template

{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true, is_last_user=false) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '

' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{ bos_token }}{{ ns.system_prompt }}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{%- set ns.is_first = false -%}{%- set ns.is_last_user = true -%}{{'<｜User｜>' + message['content'] + '<｜Assistant｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and message['tool_calls'] is defined and message['tool_calls'] is not none %}{%- set ns.is_last_user = false -%}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{%- endif %}{%- set ns.is_first = false %}{%- set ns.is_tool = false -%}{%- set ns.is_output_first = true %}{%- for tool in message['tool_calls'] %}{%- if not ns.is_first %}{%- if message['content'] is none %}{{'<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '
' + '```json' + '
' + tool['function']['arguments'] + '
' + '```' + '<｜tool▁call▁end｜>'}}{%- else %}{{message['content'] + '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '
' + '```json' + '
' + tool['function']['arguments'] + '
' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- set ns.is_first = true -%}{%- else %}{{'
' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '
' + '```json' + '
' + tool['function']['arguments'] + '
' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- endfor %}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and (message['tool_calls'] is not defined or message['tool_calls'] is none)%}{%- set ns.is_last_user = false -%}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{{content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_last_user = false -%}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'
<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_last_user and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}

Stop strings

<｜end▁of▁sentence｜>

Evidence & provenance

Source

Provenance
https://huggingface.co/deepseek-ai/DeepSeek-V3-0324
Revision pin
e9b33add76883f293d6bf61f6bd89b497e80e335
Manifest
Present

License

Name
mit
Commercial use
yes
Access
Open

How verification works →

File hashes (SHA-256)

  • deepseek-v3-0324-ud-q2-k-xl/DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf a175fb9082806ffae03c890b9a2daae46d24f39b616e0e9d9908f9b212e0ab76
  • deepseek-v3-0324-ud-q2-k-xl/DeepSeek-V3-0324-UD-Q2_K_XL-00002-of-00006.gguf 2aa680995be7c46d423515477261fabdda9cbca7cbc60bba877a58143fdb7582
  • deepseek-v3-0324-ud-q2-k-xl/DeepSeek-V3-0324-UD-Q2_K_XL-00003-of-00006.gguf 66fcda6a99d901b5fa4c5ff0eabff57af035d09ebb9f4e6ed41ad6926ef9cd0d
  • deepseek-v3-0324-ud-q2-k-xl/DeepSeek-V3-0324-UD-Q2_K_XL-00004-of-00006.gguf 63ac78c8363784885a985beff75e375c6bb63f6b31ff905b2ac5df0267f3e389
  • deepseek-v3-0324-ud-q2-k-xl/DeepSeek-V3-0324-UD-Q2_K_XL-00005-of-00006.gguf d3e6cdee637bd94744c971ffb8df34064a8a26ac0fc2e60e8321b427373a8ca6
  • deepseek-v3-0324-ud-q2-k-xl/DeepSeek-V3-0324-UD-Q2_K_XL-00006-of-00006.gguf 28c72c67d74403ea758faf93217672b0a14d2bb32d21b4bf1d53846ae7f8b536
  • deepseek-v3-0324-ud-q2-k-xl/LICENSE 1c8f573e830ca9b3ebfeb7ace1823146e22b66f99ee223840e7637c9e745e1c7
  • deepseek-v3-0324-ud-q2-k-xl/README.md 6c0954d00496da52100a57fbc15101a96a1bf18099e973e21817d95eaa5870e8

Explore further

Also consider

Get an email when a better quant fits your rig.

Set your rig once; we'll alert you when a new or better-fitting build lands.

Set your rig

Performance reports

Real-world throughput reported by the community (and scraped sources).

View performance leaderboard
Community

Reviews, sampler presets and community runtime reports. Be the first to contribute.

Reviews (0)

Log in to write a structured review of this model.

No reviews yet.

Sampler presets

Log in to share a sampler preset or vote on presets.

No presets yet.

Community runtime reports

Unofficial, community-submitted "it loads for me" reports. These are not the official verified-working badge — an admin reviews reports before anything is marked officially verified.

Log in to report whether a quant loads in your runtime.

No community reports yet.

selected to compare · pick at least 2

Compare ()