LLM Torrents

Qwen

Qwen2.5-7B-Instruct

← All models

7.6B parameters · Instruct · Qwen2.5 family

apache-2.0 source matched revision pinned

Which version should I download?

Set your rig so we can size a quant to your hardware and show which quants fit.

Set your rig

Quantizations

Format Level Size Verdict Est. speed Quality note Swarm Download
EXL2 Q4_K_M 9.81 GB Set your rig — Balanced size/quality — good default. — Not yet available
MLX Q8_0 31.35 GB Set your rig — Near-lossless; larger footprint. — Not yet available

Run it

Generic commands (no rig set). GPU-offload values assume the model fits on your GPU — set your rig for values tuned to your hardware.

Start the server 
llama-server -m qwen-qwen2-5-7b-instruct-Q4_K_M.gguf -c 8192 -ngl 999

Use llama-cli in place of llama-server for a one-shot prompt.

Save as Modelfile next to the GGUF (Modelfile) 
FROM ./qwen-qwen2-5-7b-instruct-Q4_K_M.gguf
PARAMETER num_ctx 8192
PARAMETER num_gpu 999
Create 
ollama create qwen-qwen2-5-7b-instruct -f Modelfile
Run 
ollama run qwen-qwen2-5-7b-instruct
Serve an OpenAI-compatible endpoint 
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192

Serves on http://localhost:8000/v1 by default.

Load qwen-qwen2-5-7b-instruct-Q4_K_M.gguf, set the context length to 8192 tokens. Set GPU offload to Max (all layers).

Not an MLX-format quant.

MLX runs MLX-format weights only (Apple Silicon). This quant is not an MLX build.

Filename shown is a placeholder. The exact GGUF name appears once the download is available@if ($model->license_gated) and the license is accepted.

@endif
Technical details

No template or sampling metadata recorded.

Evidence & provenance

Source

Provenance
https://huggingface.co/Qwen/Qwen2.5-7B-Instruct
Revision pin
a09a35458c702b33eeacc393d103063234e8bc28
Manifest
None

License

Name
apache-2.0
Commercial use
yes
Access
Open

File hashes (SHA-256)

No file hashes recorded yet.

Community

Performance reports

Rig Runtime Context Gen t/s VRAM Source
2x RTX 3090 llama.cpp 4,096 168.49 39.48 GB user

Reviews

  • Repellat possimus quo magnam voluptate nostrum a. Inventore laborum aut sunt consequatur voluptatibus. Necessitatibus voluptates est ut non. Rerum nobis velit tempore voluptatem.

    — Mr. Zachariah Bahringer MD

Presets

  • sint eveniet preset 12 votes · Mr. Zachariah Bahringer MD

Performance reports

Real-world throughput reported by the community (and scraped sources).

View performance leaderboard