LLM Torrents

mistralai

Mixtral-8x7B-Instruct-v0.1

← All models

46.7B parameters · mixture-of-experts · Instruct · Mixtral family

apache-2.0 source matched revision pinned

Which version should I download?

Set your rig so we can size a quant to your hardware and show which quants fit.

Set your rig

Quantizations

Format Level Size Verdict Est. speed Quality note Swarm Download
AWQ Q4_K_M 5.38 GB Set your rig — Balanced size/quality — good default. — Not yet available
safetensors Q8_0 13.60 GB Set your rig — Near-lossless; larger footprint. — Not yet available

Run it

Generic commands (no rig set). GPU-offload values assume the model fits on your GPU — set your rig for values tuned to your hardware.

Start the server 
llama-server -m mistralai-mixtral-8x7b-instruct-v0-1-Q4_K_M.gguf -c 8192 -ngl 999

Use llama-cli in place of llama-server for a one-shot prompt.

Save as Modelfile next to the GGUF (Modelfile) 
FROM ./mistralai-mixtral-8x7b-instruct-v0-1-Q4_K_M.gguf
PARAMETER num_ctx 8192
PARAMETER num_gpu 999
Create 
ollama create mistralai-mixtral-8x7b-instruct-v0-1 -f Modelfile
Run 
ollama run mistralai-mixtral-8x7b-instruct-v0-1
Serve an OpenAI-compatible endpoint 
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 --max-model-len 8192 --quantization awq

Serves on http://localhost:8000/v1 by default.

Load mistralai-mixtral-8x7b-instruct-v0-1-Q4_K_M.gguf, set the context length to 8192 tokens. Set GPU offload to Max (all layers).

Not an MLX-format quant.

MLX runs MLX-format weights only (Apple Silicon). This quant is not an MLX build.

Filename shown is a placeholder. The exact GGUF name appears once the download is available@if ($model->license_gated) and the license is accepted.

@endif
Technical details

No template or sampling metadata recorded.

Evidence & provenance

Source

Provenance
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
Revision pin
eba92302a2861cdc0098cc54bc9f17cb2c47eb61
Manifest
None

License

Name
apache-2.0
Commercial use
yes
Access
Open

File hashes (SHA-256)

No file hashes recorded yet.

Community

Performance reports

Rig Runtime Context Gen t/s VRAM Source
RTX 4090 llama.cpp 32,768 18.52 75.72 GB user

Reviews

  • Vitae eius praesentium sit quia. Ex voluptatibus deleniti est sint est. Et quam et velit laborum vero enim autem. Recusandae porro excepturi dignissimos.

    — Mr. Zachariah Bahringer MD

Presets

  • enim nostrum preset 23 votes · Mr. Zachariah Bahringer MD

Performance reports

Real-world throughput reported by the community (and scraped sources).

View performance leaderboard