Best local LLMs for 96 GB of VRAM or unified memory
Measured speeds from builds where the model ran on more than 64 and up to 96 GB of GPU or unified memory. Each number links to its build and source.
8 GB · 12 GB · 16 GB · 24 GB · 32 GB · 48 GB · 64 GB · 96 GB · 128 GB or more
GPUs in this tier: RTX Pro 6000 Blackwell
Fastest measured result per model
| Model | Plain decode tok/s | With speculative decoding ⚡ | Results |
|---|---|---|---|
| Qwen3 235B-A22B | 18 (IQ4_XS, 4x RTX 3090) | – | 2 |
| Qwen3.8 27B | – | 150 (NVFP4 (W4A16) + MTP (5 draft tokens), 256K context, RTX Pro 6000 Max-Q (Blackwell, 96 GB, 300 W)) | 2 |
| gpt-oss 120B | 196.31 (MXFP4, RTX Pro 6000 (Blackwell, 96 GB)) | – | 1 |
| gpt-oss 20B | 286.91 (MXFP4, RTX Pro 6000 (Blackwell, 96 GB)) | – | 1 |
| Qwen3.6 35B-A3B | 50.2-62.4 (llama-bench tg128 (quant per result), Strix Halo (Radeon 8060S)) | – | 1 |
Excludes results the source marks pending and results where part of the model sat in system RAM.
All results in this tier
(MTP, DFlash, DSpark and similar draft-and-verify methods give much higher tok/s on predictable text and are not comparable 1:1 with plain decode)
| Model | Build | Ran on | Quant | Engine | tok/s | Details |
|---|---|---|---|---|---|---|
| Qwen3 235B-A22B | 4x RTX 3090 + EPYC 7443, 256 GB DDR4 8-channel ⚠ unchecked | 4x RTX 3090 (96 GB) | IQ4_XS | llama.cpp | 18 | |
| Qwen3 235B-A22B | 4x RTX 3090 + 32-core Threadripper, 192 GB DDR4 ⚠ unchecked | 4x RTX 3090 (96 GB) | UD-Q3_K_XL | llama.cpp | 12-14 | |
| Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S, 96 GB): 2026 models (llamabench community) | Strix Halo (Radeon 8060S) (96 GB) | llama-bench tg128 (quant per result) | llama.cpp | 50.2-62.4 | |
| Qwen3.8 27B | RTX Pro 6000 Blackwell Max-Q (300 W cap): Qwen3.8 27B on vLLM, NVFP4 vs BF16 | RTX Pro 6000 Max-Q (Blackwell, 96 GB, 300 W) (96 GB) | NVFP4 (W4A16) + MTP (5 draft tokens), 256K context | vLLM 0.27.1 | 150 | ⚡ spec-decode (MTP) |
| Qwen3.8 27B | RTX Pro 6000 Blackwell Max-Q (300 W cap): Qwen3.8 27B on vLLM, NVFP4 vs BF16 | RTX Pro 6000 Max-Q (Blackwell, 96 GB, 300 W) (96 GB) | BF16 + MTP (5 draft tokens), 256K context | vLLM 0.20.1 | 85 | ⚡ spec-decode (MTP) |
| gpt-oss 120B | RTX Pro 6000 96GB workstation card | RTX Pro 6000 (Blackwell, 96 GB) (96 GB) | MXFP4 | llama.cpp | 196.31 | |
| gpt-oss 20B | RTX Pro 6000 96GB workstation card | RTX Pro 6000 (Blackwell, 96 GB) (96 GB) | MXFP4 | llama.cpp | 286.91 |
What fits by memory math
Q4_K_M weights (4.8944 bits per weight, per the llama.cpp quantize table) that fit in 96 GB with ~5% headroom, before KV cache: Mistral Medium 3.5 128B ≈78 GB, Ornith 1.5 122B ≈77 GB, Qwen3.5 122B ≈75 GB, Nemotron 3 Super 120B-A12B ≈73 GB, Nemotron 3.5 Super VL 120B-A12B ≈73 GB, gpt-oss 120B ≈72 GB, Nemotron 3.5 Bolt 74B-A9B ≈45 GB, Llama 3.3 70B ≈43 GB, Flint 40B-A2.5B ≈24 GB, Ornith 1.5 35B-A3B ≈22 GB, Qwen3.6 35B-A3B ≈22 GB, Gemma 4 31B ≈19 GB, Nemotron 3.5 Lightning 30B-A3B ≈18 GB, Qwen3.6 27B ≈17 GB, Qwen3.8 27B ≈17 GB, Gemma 4 26B-A4B ≈15 GB, Devstral Small 2 24B ≈15 GB, LFM2.5 24B-A2B ≈15 GB, gpt-oss 20B ≈13 GB, Gemma 4 12B ≈7 GB, Qwen3.5 9B ≈5 GB, Llama 3.1 8B ≈5 GB.
FAQ
Which local LLMs have been measured on 96 GB of VRAM?
Qwen3 235B-A22B, gpt-oss 120B, gpt-oss 20B, Qwen3.6 35B-A3B, Qwen3.8 27B (7 results).
What fits in 96 GB of VRAM?
By memory math at Q4_K_M (4.8944 bits per weight, llama.cpp), weights only, before KV cache and context: Mistral Medium 3.5 128B (≈78 GB), Ornith 1.5 122B (≈77 GB), Qwen3.5 122B (≈75 GB), Nemotron 3 Super 120B-A12B (≈73 GB), Nemotron 3.5 Super VL 120B-A12B (≈73 GB), gpt-oss 120B (≈72 GB), Nemotron 3.5 Bolt 74B-A9B (≈45 GB), Llama 3.3 70B (≈43 GB). Leave a few GB for context.
Are speculative-decoding results comparable?
Not 1:1. MTP, DFlash and similar methods draft tokens that the main model verifies, which can double tok/s on predictable text. This page lists them in a separate column; tick the box to hide them in the full table.
Referral link: if you sign up with Google and add $10, Runpod gives you and RigBase bonus credit (terms). It never affects our data.