moonshotai

Kimi-VL-A3B-Instruct

Name: Kimi-VL-A3B-Instruct
Author: moonshotai

16.4B parameters · mixture-of-experts · Instruct · Kimi family

Mixture-of-experts — what it means for memory & speed → Long context — how it uses memory →

mit hash verified source matched revision pinned

Which version should I download?

Default pick: GGUF · Q4_K_M — mradermacher Q4_K_M quant of Kimi-VL-A3B-Instruct (16.408B-A3B vision-language MoE) — recommended balanced pick; bundled mmproj-Q8_0 required for vision. MIT.

Set your rig to confirm it fits and to see if a higher-quality quant runs fully on your hardware.

Set your rig

Quantizations

Why these quants? Which quant should I pick? What do these mean?

Format	Level	Size	Verdict	Est. speed	Quality note	Swarm	Download
GGUF	IQ4_XS	8.15 GB	Set your rig	—	mradermacher IQ4_XS quant of Kimi-VL-A3B-Instruct (16.408B-A3B vision-language MoE) — compact low-bit option; bundled mmproj-Q8_0 required for vision. MIT.	2S / 0L webseed OK	Magnet .torrent
GGUF	Q4_K_M	9.82 GB	Set your rig	—	mradermacher Q4_K_M quant of Kimi-VL-A3B-Instruct (16.408B-A3B vision-language MoE) — recommended balanced pick; bundled mmproj-Q8_0 required for vision. MIT.	2S / 0L webseed OK	Magnet .torrent
GGUF	Q8_0	15.81 GB	Set your rig	—	mradermacher Q8_0 quant of Kimi-VL-A3B-Instruct (16.408B-A3B vision-language MoE) — high-quality option; bundled mmproj-f16 required for vision. MIT.	2S / 0L webseed OK	Magnet .torrent
safetensors	BF16	30.57 GB	Set your rig	—	Official Moonshot BF16 safetensors — full Kimi-VL-A3B-Instruct package (16.408B-A3B VL MoE + MoonViT vision encoder, tokenizer + modeling code): source-format artifact for fine-tuning/conversion. Full-capability package. MIT.	2S / 0L webseed OK	Magnet .torrent

GGUF · IQ4_XS

8.15 GB

Set your rig

Est. speed: —
Swarm: 2S / 0L webseed OK

mradermacher IQ4_XS quant of Kimi-VL-A3B-Instruct (16.408B-A3B vision-language MoE) — compact low-bit option; bundled mmproj-Q8_0 required for vision. MIT.

Magnet .torrent

GGUF · Q4_K_M

9.82 GB

Set your rig

Est. speed: —
Swarm: 2S / 0L webseed OK

mradermacher Q4_K_M quant of Kimi-VL-A3B-Instruct (16.408B-A3B vision-language MoE) — recommended balanced pick; bundled mmproj-Q8_0 required for vision. MIT.

Magnet .torrent

GGUF · Q8_0

15.81 GB

Set your rig

Est. speed: —
Swarm: 2S / 0L webseed OK

mradermacher Q8_0 quant of Kimi-VL-A3B-Instruct (16.408B-A3B vision-language MoE) — high-quality option; bundled mmproj-f16 required for vision. MIT.

Magnet .torrent

safetensors · BF16

30.57 GB

Set your rig

Est. speed: —
Swarm: 2S / 0L webseed OK

Official Moonshot BF16 safetensors — full Kimi-VL-A3B-Instruct package (16.408B-A3B VL MoE + MoonViT vision encoder, tokenizer + modeling code): source-format artifact for fine-tuning/conversion. Full-capability package. MIT.

Magnet .torrent

Want a different quant? Request it on the board →

Runs fully on

RTX 5090 · Q8_0 RTX 5060 Ti 16GB · Q4_K_M RTX 5070 Ti · Q4_K_M RTX 5080 · Q4_K_M RTX 5070 · Q4_K_M

Green = the model's best quant fits fully in GPU/unified memory at 8K context. Tap a card for its full "what runs on it" page.

Context

Advertised 128k · no independent evidence yet

No independent long-context evidence has been graded for this model yet — the advertised window above is the maintainer's number, not a usable-context claim.

Capabilities (as declared by the maintainer): tool calling

Run it

Runtime completeness (IQ4_XS torrent): llama.cpp ✅ Ollama – vision sidecar –

– Ollama — runs the GGUF directly; no Modelfile bundled
– vision sidecar — not a vision model

Quant

Context / KV 8,192 tokens · FP16 set above the quant table

Generic commands (no rig set). GPU-offload values assume the model fits on your GPU — set your rig for values tuned to your hardware.

Start the server

llama-server -m Kimi-VL-A3B-Instruct.i1-IQ4_XS.gguf -c 8192 -ngl 999

Use llama-cli in place of llama-server for a one-shot prompt. This model compresses its KV cache, but only on llama.cpp b5137 (April 2025) or newer AND a GGUF converted after that date. Older files -- including some still-popular DeepSeek-R1 and V3-0324 quants -- fall back to the uncompressed layout and use about 5 MB per token instead of 70 KB. Check the quant repo date if long context is the point.

Save as Modelfile next to the GGUF (Modelfile)

FROM ./Kimi-VL-A3B-Instruct.i1-IQ4_XS.gguf
PARAMETER num_ctx 8192
PARAMETER num_gpu 999
PARAMETER stop "[EOS]"
PARAMETER temperature 0.2
# Chat template: Ollama uses the template embedded in the GGUF (no TEMPLATE directive needed).

Create

ollama create moonshotai-kimi-vl-a3b-instruct -f Modelfile

Run

ollama run moonshotai-kimi-vl-a3b-instruct

GGUF is not a first-class vLLM format.

vLLM GGUF support is experimental and single-file only; prefer safetensors/GPTQ/AWQ for production. If you must, pass the .gguf path to `vllm serve` with --load-format gguf on a recent vLLM.

transformers does not load GGUF weights.

GGUF weights run under llama.cpp, Ollama or LM Studio.

Load Kimi-VL-A3B-Instruct.i1-IQ4_XS.gguf, set the context length to 8192 tokens. Set GPU offload to Max (all layers).

MLX runs MLX-format weights only (Apple Silicon). This quant is a GGUF build.

GGUF weights run under llama.cpp, Ollama or LM Studio.

Technical details

Chat template

{%- for message in messages -%}{%- if loop.first and messages[0]['role'] != 'system' -%}{{'<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|>'}}{%- endif -%}{%- if message['role'] == 'system' -%}{{'<|im_system|>'}}{%- endif -%}{%- if message['role'] == 'user' -%}{{'<|im_user|>'}}{%- endif -%}{%- if message['role'] == 'assistant' -%}{{'<|im_assistant|>'}}{%- endif -%}{{- message['role'] -}}{{'<|im_middle|>'}}{%- if message['content'] is string -%}{{- message['content'] + '<|im_end|>' -}}{%- else -%}{%- for content in message['content'] -%}{%- if content['type'] == 'image' or 'image' in content or 'image_url' in content -%}{{'<|media_start|>image<|media_content|><|media_pad|><|media_end|>'}}{%- else -%}{{content['text']}}{%- endif -%}{%- endfor -%}{{'<|im_end|>'}}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{'<|im_assistant|>assistant<|im_middle|>'}}{%- endif -%}

Sampling defaults

temperature: 0.2

Stop strings

[EOS]

Evidence & provenance

Source

Provenance: https://huggingface.co/moonshotai/Kimi-VL-A3B-Instruct
Revision pin: 398eede0903cd983a2bfa0cc634e9ac1d843f375
Manifest: Present

License

Name: mit
Commercial use: yes
Access: Open

How verification works →

File hashes (SHA-256)

kimi-vl-a3b-instruct-iq4-xs/Kimi-VL-A3B-Instruct.i1-IQ4_XS.gguf 5d6aad466030edeb73f44b4a32c0c98d081d0e03a300366d6b63fbf469bff073
kimi-vl-a3b-instruct-iq4-xs/Kimi-VL-A3B-Instruct.mmproj-Q8_0.gguf 0b58fc0e75e3aba661b0f3a5db23ab3ca33c33a38c9078d3c200bfbc681e2916
kimi-vl-a3b-instruct-iq4-xs/LICENSE f0c89806155f93f4ea5cb9e81ef70aad1a7729240132df9706db9584024547db
kimi-vl-a3b-instruct-iq4-xs/README.md 7fa4d4e39537a43a7d6cc6f62873eb014b2f8bc3da2142111ba2ad29d4984794
kimi-vl-a3b-instruct-q4-k-m/Kimi-VL-A3B-Instruct.i1-Q4_K_M.gguf 7710f9aa5bf56a38a03f26738cdbf10563a72938a41025307b57768dad40f306
kimi-vl-a3b-instruct-q4-k-m/Kimi-VL-A3B-Instruct.mmproj-Q8_0.gguf 0b58fc0e75e3aba661b0f3a5db23ab3ca33c33a38c9078d3c200bfbc681e2916
kimi-vl-a3b-instruct-q4-k-m/LICENSE f0c89806155f93f4ea5cb9e81ef70aad1a7729240132df9706db9584024547db
kimi-vl-a3b-instruct-q4-k-m/README.md 7fa4d4e39537a43a7d6cc6f62873eb014b2f8bc3da2142111ba2ad29d4984794
kimi-vl-a3b-instruct-q8-0/Kimi-VL-A3B-Instruct.Q8_0.gguf a13e350588914efde35c9fc9935e0bbc26f0aa81066f53929a381743a509ca67
kimi-vl-a3b-instruct-q8-0/Kimi-VL-A3B-Instruct.mmproj-f16.gguf 794ffead89866df5b3c698c74ca16bd7d31b48575123edee24f476c9a15c4ad5
kimi-vl-a3b-instruct-q8-0/LICENSE f0c89806155f93f4ea5cb9e81ef70aad1a7729240132df9706db9584024547db
kimi-vl-a3b-instruct-q8-0/README.md 7fa4d4e39537a43a7d6cc6f62873eb014b2f8bc3da2142111ba2ad29d4984794
kimi-vl-a3b-instruct/.gitattributes c58cb09727b7fea047dc29f253c6aa3632fc93b538d055cb8db9831db024d73f
kimi-vl-a3b-instruct/LICENSE f0c89806155f93f4ea5cb9e81ef70aad1a7729240132df9706db9584024547db
kimi-vl-a3b-instruct/README.md 7fa4d4e39537a43a7d6cc6f62873eb014b2f8bc3da2142111ba2ad29d4984794
kimi-vl-a3b-instruct/arch.png 5195d9f99c08f7e135eedb19cf370d92c36b7b3387e9c1b7cad5e24990a0d6d0
kimi-vl-a3b-instruct/chat_template.jinja 172c7450a66b9b3644538888ffa24d02047e535e9d1f911564460203fbf7e8e8
kimi-vl-a3b-instruct/config.json af3810295de6b08232776b74110ed482fa0183f4aeaec586ff4f7593910951cb
kimi-vl-a3b-instruct/configuration_kimi_vl.py d30462b89993b9fef138db983d3884bbc39750effadf2efe67f404d28ee40e5b
kimi-vl-a3b-instruct/demo.png 95de8765da89c41a2421f1c1fa3986e4d3c83793d92c8ade460a142b329d04c1
kimi-vl-a3b-instruct/generation_config.json da26dd1916774784790db2b83336bad62d0917356743478ba6eb0c425c53eac5
kimi-vl-a3b-instruct/image_processing_kimi_vl.py bb58551e088f3bb0027b1886ca5ff42776c485acec1a9aafe91553ff76926870
kimi-vl-a3b-instruct/instruct_perf.png 52405bbe3e3b0c30a5502c40095241000d4c3dbf9862d7a6c0b6079806292ad4
kimi-vl-a3b-instruct/logo.png 7870b48105beb49cdb29bb3090abb7bbca688bef862507904c23d9c472df221c
kimi-vl-a3b-instruct/model-00001-of-00007.safetensors a5ef3ebd9727f82e34417a778317d2cc9c08762fe0bc4a2ee333b8a52cf7c1a5
kimi-vl-a3b-instruct/model-00002-of-00007.safetensors 45ecd00decdad65e7d3f494028ed0c79d0cd56f145adae077ffa78b5b8ff95c0
kimi-vl-a3b-instruct/model-00003-of-00007.safetensors b9ba98f01e22eea43da8cfbf6f09ff0857616cd9e1df4603a735c111755109ef
kimi-vl-a3b-instruct/model-00004-of-00007.safetensors 8f7eb3fc5c12481fd1a81d2708fa4a299ff4c207a5dbafb5b8bef25ab9fd8b23
kimi-vl-a3b-instruct/model-00005-of-00007.safetensors db83e896b3d75f4bc51f621ae90ad61e8f6f7901f49a06b5c40348b839057206
kimi-vl-a3b-instruct/model-00006-of-00007.safetensors abed41982a3f9c7d05f69bb560dad7bcb93cfa764c77a2b59127f35dc787983c
kimi-vl-a3b-instruct/model-00007-of-00007.safetensors 2e2ddd33f2b4f472898482860585bd6d73d4397c8c833ed9d00a2024443f7a77
kimi-vl-a3b-instruct/model.safetensors.index.json 178ba3a15cb12f6ceac42aecfe665a8f041d47e948220d75b073088957efedd8
kimi-vl-a3b-instruct/modeling_kimi_vl.py 74e8f18afa5bf8f580ef5be5f15b51506f1e4f604df5a84f508a7796b8947c0b
kimi-vl-a3b-instruct/preprocessor_config.json d968a7d7f83f82c3b0e1061079dcdbefd6f2c75b7ee4240cce995afaeae1f622
kimi-vl-a3b-instruct/processing_kimi_vl.py f8e64816baed0a92d85d0a987b587028ad55a0779f3ace8c5b68db705dc61561
kimi-vl-a3b-instruct/tiktoken.model b6c497a7469b33ced9c38afb1ad6e47f03f5e5dc05f15930799210ec050c5103
kimi-vl-a3b-instruct/tokenization_moonshot.py 0ae85e29c37689fefe41b8bfdb7aebd9a0152defedbf37fae0f8413000512029
kimi-vl-a3b-instruct/tokenizer_config.json d7861718cab22c7962e14b4693b4009dd526115cd392dc779069a8173be95703

Explore further

Appears in

What runs on 12 GB of VRAM What runs on 16 GB of VRAM What runs on a 16 GB Mac What runs on a 24 GB Mac Coding models that fit 12 GB Writing models that fit 12 GB

Also consider

Get an email when a better quant fits your rig.

Set your rig once; we'll alert you when a new or better-fitting build lands.

Set your rig

Performance reports

Real-world throughput reported by the community (and scraped sources).

View performance leaderboard

Community

Reviews, sampler presets and community runtime reports. Be the first to contribute.

Reviews (0)

No reviews yet.

Sampler presets

No presets yet.

Community runtime reports

Unofficial, community-submitted "it loads for me" reports. These are not the official verified-working badge — an admin reviews reports before anything is marked officially verified.

No community reports yet.

Recommended download

Quantizations

Runs fully on

Context

Run it

Chat template

Sampling defaults

Stop strings

Evidence & provenance

Source

License

File hashes (SHA-256)

Explore further

Appears in

Also consider

Performance reports

Reviews (0)

Sampler presets

Community runtime reports