Best home rig for Mistral Medium 3.5 128B

Will it fit my GPU? Best value GPU per euro →

Real, measured generation speeds for Mistral Medium 3.5 128B on home hardware, fastest first. Different quants/engines aren't directly comparable.

Build	Quant	Engine	tok/s
4x Mac Studio M5 Ultra cluster (Thunderbolt 5 RDMA): Muse Spark Open at 36.7 tok/s	FP8, 8,192-token context	MLX, 4-node tensor parallel	23.37	source
2x Mac Studio M5 Ultra cluster (Thunderbolt 5 RDMA): Muse Spark Open 816B	FP8, 8,192-token context	MLX, 2-node tensor parallel	13.69	source
Mac Studio M5 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark)	FP8, 8,192-token context	MLX (mlx_lm.benchmark)	7.7	source
Mac Studio M3 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark)	FP8, 8,192-token context	MLX (mlx_lm.benchmark)	4.95	source

Rent before you buy: try this model on a cloud GPU for a few dollars first: Runpod (referral link; sign up with Google to get the new-user bonus)

Memory math: how much VRAM / RAM does Mistral Medium 3.5 128B need?

Mistral Medium 3.5 128B has 128B parameters (source: Ollama library: 128B weights). Weight size = parameters × bits-per-weight ÷ 8, before context (KV cache) and runtime overhead.

Format	Bits/weight	Weights	Smallest common memory size that holds the weights alone
Q8_0	8.50	136.0 GB	192 GB
Q6_K	6.56	105.0 GB	128 GB
Q5_K_M	5.70	91.3 GB	96 GB
Q4_K_M	4.89	78.3 GB	96 GB
Q3_K_M	4.00	63.9 GB	64 GB
Q2_K	3.16	50.5 GB	64 GB

Bits-per-weight for GGUF quants are llama.cpp's own figures (llama.cpp quantize README). "As measured" rows apply the source's stated bits-per-weight to the parameter count; real files differ slightly because some layers stay at higher precision. Try other combinations in the VRAM calculator.

FAQ

How much VRAM does Mistral Medium 3.5 128B need?

The weights are about 78.3 GB at Q4_K_M (128B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is Apple M5 Ultra (512 GB) at 7.7 tok/s.

What is the fastest measured hardware for Mistral Medium 3.5 128B?

4x Apple M5 Ultra: 23.37 tok/s generation (MLX, 4-node tensor parallel, FP8, 8,192-token context), from https://machinelearning.apple.com/research/mlx-distributed-inference.

Which GPUs have measured results for Mistral Medium 3.5 128B?

2x Apple M5 Ultra, 4x Apple M5 Ultra, Apple M3 Ultra, Apple M5 Ultra.

Can it run on… 2x Apple M5 Ultra · 4x Apple M5 Ultra · Apple M3 Ultra · Apple M5 Ultra