Best local LLMs for 24 GB of VRAM or unified memory

Measured speeds from builds where the model ran on more than 16 and up to 24 GB of GPU or unified memory. Each number links to its build and source.

8 GB · 12 GB · 16 GB · 24 GB · 32 GB · 48 GB · 64 GB · 96 GB · 128 GB or more

GPUs in this tier: RTX 3090 · RTX 4090 · Radeon RX 7900 XTX

Fastest measured result per model

Model	Plain decode tok/s	With speculative decoding ⚡	Results
Qwen3.6 35B-A3B	160.6 (Q4_K_M, RTX 4090)	135.2 (UD-Q4_K_XL + MTP (draft-n-max 4), ~16K context, Radeon RX 7900 XTX)	10
Qwen3.8 27B	42.9 (Q4_K_M (unsloth), no speculation, RTX 3090)	139.7 (Q4_K_M (unsloth), greedy code prompt, RTX 3090)	7
gpt-oss 20B	279.65 (, RTX 4090)	–	4
Llama 3.2 3B	334.67 (, RTX 4090)	–	3
Qwen3.6 27B	47.0 (llama-bench tg128 (quant per result), RTX 4090)	–	3
Flint 40B-A2.5B	179.4 (NVFP4, RTX 4090)	–	3
Gemma 4 12B	98.1 (llama-bench tg128 (quant per result), RTX 4090)	–	2
GLM 4.7 Flash	164.9 (Q4_K_M (EPYC 75F3 host), RTX 4090)	–	2
Llama 2 7B	188.96 (Q4_0, RTX 4090)	–	1
Llama 3.1 8B	52.99 (Q8, RTX 4090)	–	1
Mistral Small 22B	26.72 (Q6_K, RTX 4090)	–	1
DeepSeek R1 671B	–	–	1
DeepSeek R1 Distill 14B	92.49 (, RTX 4090)	–	1
Llama 3.1 70B	3.1 (, RTX 4090)	–	1
Gemma 4 31B	43.8 (llama-bench tg128 (quant per result), RTX 4090)	–	1
Gemma 4 26B-A4B	164 (llama-bench tg128 (quant per result), RTX 4090)	–	1
Qwen3.5 9B	137 (llama-bench tg128 (quant per result), RTX 4090)	–	1
LFM2.5 24B-A2B	261-339 (llama-bench tg128 (quant per result), RTX 4090)	–	1
Nemotron 3.5 Bolt 74B-A9B	–	–	1

Excludes results the source marks pending and results where part of the model sat in system RAM.

All results in this tier

Hide speculative-decoding results ⚡ (MTP, DFlash, DSpark and similar draft-and-verify methods give much higher tok/s on predictable text and are not comparable 1:1 with plain decode)

Model	Build	Ran on	Quant	Engine	tok/s	Details
DeepSeek R1 671B	EPYC + single RTX 3090 (hybrid) ⚠ unchecked	RTX 3090 (24 GB)	Q2_K	llama.cpp CUDA	8.97	partly in system RAM
DeepSeek R1 Distill 14B	Intel Core Ultra 265K + RTX 4090	RTX 4090 (24 GB)		llama.cpp	92.49
Flint 40B-A2.5B	RTX 4090 (llm-speed signed run, Oct 2026): Flint 40B-A2.5B at 169-179 tok/s on llama.cpp	RTX 4090 (24 GB)	NVFP4	llama.cpp 0.7.0-dev (build 11803)	179.4
Flint 40B-A2.5B	RTX 4090 (llm-speed signed run, Oct 2026): Flint 40B-A2.5B at 169-179 tok/s on llama.cpp	RTX 4090 (24 GB)	UD-Q3_K_XL	llama.cpp 0.7.0-dev (build 11803)	169.2
Flint 40B-A2.5B	RTX 3090 (llm-speed signed run, Oct 2026): Flint 40B-A2.5B Q3_K_XL at 128.9 tok/s	RTX 3090 (24 GB)	UD-Q3_K_XL	llama.cpp 0.7.0-dev (build 11803)	128.9
GLM 4.7 Flash	RTX 4090 (llm-speed signed runs, Oct 2026): Qwen3.6 35B-A3B 161 tok/s, Gemma 4 12B QAT 98 tok/s	RTX 4090 (24 GB)	Q4_K_M (EPYC 75F3 host)	Ollama 0.31.1	164.9
GLM 4.7 Flash	RTX 3090 (llm-speed signed runs, Oct 2026): gpt-oss 20B 207 tok/s, Qwen3.6 27B 43 tok/s	RTX 3090 (24 GB)	Q4_K_M (Threadripper PRO host)	Ollama 0.31.1	134.6
Gemma 4 12B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	98.1
Gemma 4 12B	RTX 4090 (llm-speed signed runs, Oct 2026): Qwen3.6 35B-A3B 161 tok/s, Gemma 4 12B QAT 98 tok/s	RTX 4090 (24 GB)	QAT Q4_0	llama.cpp b8931	97.7
Gemma 4 26B-A4B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	164
Gemma 4 31B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	43.8
LFM2.5 24B-A2B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	261-339
Llama 2 7B	Single RTX 4090 (llama.cpp CUDA scoreboard)	RTX 4090 (24 GB)	Q4_0	llama.cpp CUDA (FA)	188.96
Llama 3.1 70B	Ryzen 9 7900X + RTX 4090	RTX 4090 (24 GB)		GPU+CPU offload	3.1
Llama 3.1 8B	RTX 4090 desktop (speed comparison vs Macs)	RTX 4090 (24 GB)	Q8	llama.cpp-based	52.99
Llama 3.2 3B	Intel Core Ultra 265K + RTX 4090	RTX 4090 (24 GB)		llama.cpp	334.67
Llama 3.2 3B	Ryzen 9 7900X + RTX 4090	RTX 4090 (24 GB)		llama.cpp	237.05
Llama 3.2 3B	Ryzen 9 9950X + Radeon RX 7900 XT	Radeon RX 7900 XT (20 GB)		llama.cpp	131.2
Mistral Small 22B	RTX 4090 desktop (speed comparison vs Macs)	RTX 4090 (24 GB)	Q6_K	llama.cpp-based	26.72
Nemotron 3.5 Bolt 74B-A9B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	108	partly in system RAM
Qwen3.5 9B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	137
Qwen3.6 27B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	47.0
Qwen3.6 27B	RTX 3090 (llm-speed signed runs, Oct 2026): gpt-oss 20B 207 tok/s, Qwen3.6 27B 43 tok/s	RTX 3090 (24 GB)	Q4_K_M	llama.cpp 0.7.0-dev	42.7
Qwen3.6 27B	RTX 3090: 2026 models (llamabench community)	RTX 3090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	40.2
Qwen3.6 35B-A3B	RTX 4090 (llm-speed signed runs, Oct 2026): Qwen3.6 35B-A3B 161 tok/s, Gemma 4 12B QAT 98 tok/s	RTX 4090 (24 GB)	Q4_K_M	llama.cpp b8931	160.6
Qwen3.6 35B-A3B	RTX 3090: 2026 models (llamabench community)	RTX 3090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	146-153
Qwen3.6 35B-A3B	Single RTX 3090: Qwen3.6 35B-A3B (Giles Thomas)	RTX 3090 (24 GB)	Q4 GGUF, all layers on GPU, ~90K ctx	llama.cpp CUDA	139.6
Qwen3.6 35B-A3B	Radeon RX 7900 XTX (24 GB): Qwen3.6 35B-A3B up to 245K context, with MTP	Radeon RX 7900 XTX (24 GB)	UD-Q4_K_XL + MTP (draft-n-max 4), ~16K context	llama.cpp ROCm (llama-server)	135.2	⚡ spec-decode (MTP)
Qwen3.6 35B-A3B	4x RTX 3090 lab: Qwen3.6 on vLLM vs llama.cpp	RTX 3090 (24 GB)	IQ4_XS (1 GPU)	llama.cpp	115-133
Qwen3.6 35B-A3B	Radeon RX 7900 XTX (24 GB): Qwen3.6 35B-A3B up to 245K context, with MTP	Radeon RX 7900 XTX (24 GB)	UD-Q4_K_XL + MTP, ~64K context	llama.cpp ROCm (llama-server)	108.2	⚡ spec-decode (MTP)
Qwen3.6 35B-A3B	Radeon RX 7900 XTX (24 GB): Qwen3.6 35B-A3B up to 245K context, with MTP	Radeon RX 7900 XTX (24 GB)	UD-Q4_K_XL, llama-bench 2K, f16 KV	llama.cpp ROCm	87.2
Qwen3.6 35B-A3B	Radeon RX 7900 XTX (24 GB): Qwen3.6 35B-A3B up to 245K context, with MTP	Radeon RX 7900 XTX (24 GB)	UD-Q4_K_XL, ~16K real-text context, q4_0 KV	llama.cpp ROCm (llama-server)	79.6
Qwen3.6 35B-A3B	Radeon RX 7900 XTX (24 GB): Qwen3.6 35B-A3B up to 245K context, with MTP	Radeon RX 7900 XTX (24 GB)	UD-Q4_K_XL, ~64K context	llama.cpp ROCm (llama-server)	66.0
Qwen3.6 35B-A3B	Radeon RX 7900 XTX (24 GB): Qwen3.6 35B-A3B up to 245K context, with MTP	Radeon RX 7900 XTX (24 GB)	UD-Q4_K_XL, ~245K context, q4_0 KV	llama.cpp ROCm (llama-server)	38.3
Qwen3.8 27B	Single RTX 3090: Qwen3.8 27B at 78-140 tok/s with the OpenJet megakernel vs llama.cpp	RTX 3090 (24 GB)	Q4_K_M (unsloth), greedy code prompt	OpenJet megakernel (MTP, 4 drafts)	139.7	⚡ spec-decode (MTP)
Qwen3.8 27B	Single RTX 3090: Qwen3.8 27B at 78-140 tok/s with the OpenJet megakernel vs llama.cpp	RTX 3090 (24 GB)	Q4_K_M (unsloth), reasoning mode	OpenJet megakernel (MTP)	78.0	⚡ spec-decode (MTP)
Qwen3.8 27B	Single RTX 3090: Qwen3.8 27B at 78-140 tok/s with the OpenJet megakernel vs llama.cpp	RTX 3090 (24 GB)	Q4_K_M (unsloth), greedy code prompt	llama.cpp (MTP, 2 drafts)	72.5	⚡ spec-decode (MTP)
Qwen3.8 27B	Single RTX 3090: Qwen3.8 27B at 78-140 tok/s with the OpenJet megakernel vs llama.cpp	RTX 3090 (24 GB)	Q4_K_M (unsloth), reasoning mode	llama.cpp (MTP, 2 drafts)	50.9	⚡ spec-decode (MTP)
Qwen3.8 27B	Single RTX 3090: Qwen3.8 27B at 78-140 tok/s with the OpenJet megakernel vs llama.cpp	RTX 3090 (24 GB)	Q4_K_M (unsloth), no speculation	OpenJet megakernel	42.9
Qwen3.8 27B	Single RTX 3090: Qwen3.8 27B at 78-140 tok/s with the OpenJet megakernel vs llama.cpp	RTX 3090 (24 GB)	Q4_K_M (unsloth), no speculation	llama.cpp	40.4
Qwen3.8 27B	RTX 3090: 2026 models (llamabench community)	RTX 3090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	38.2-71.6
gpt-oss 20B	Intel Core Ultra 265K + RTX 4090	RTX 4090 (24 GB)		llama.cpp	279.65
gpt-oss 20B	RTX 3090 (llm-speed signed runs, Oct 2026): gpt-oss 20B 207 tok/s, Qwen3.6 27B 43 tok/s	RTX 3090 (24 GB)	reported as Q4_K_M	llama.cpp 0.7.0-dev	207.2
gpt-oss 20B	RTX 4090: 2026 models (llamabench community)	RTX 4090 (24 GB)	llama-bench tg128 (quant per result)	llama.cpp	203
gpt-oss 20B	Single RTX 3090 24GB	RTX 3090 (24 GB)	MXFP4	llama.cpp	161.77

What fits by memory math

Q4_K_M weights (4.8944 bits per weight, per the llama.cpp quantize table) that fit in 24 GB with ~5% headroom, before KV cache: Ornith 1.5 35B-A3B ≈22 GB, Qwen3.6 35B-A3B ≈22 GB, Gemma 4 31B ≈19 GB, Nemotron 3.5 Lightning 30B-A3B ≈18 GB, Qwen3.6 27B ≈17 GB, Qwen3.8 27B ≈17 GB, Gemma 4 26B-A4B ≈15 GB, Devstral Small 2 24B ≈15 GB, LFM2.5 24B-A2B ≈15 GB, gpt-oss 20B ≈13 GB, Gemma 4 12B ≈7 GB, Qwen3.5 9B ≈5 GB, Llama 3.1 8B ≈5 GB.

FAQ

Which local LLMs have been measured on 24 GB of VRAM?

Llama 2 7B, Llama 3.1 8B, Mistral Small 22B, DeepSeek R1 671B, gpt-oss 20B, DeepSeek R1 Distill 14B, Llama 3.2 3B, Llama 3.1 70B, Qwen3.6 27B, Gemma 4 31B, Gemma 4 26B-A4B, Gemma 4 12B, Qwen3.5 9B, LFM2.5 24B-A2B, Nemotron 3.5 Bolt 74B-A9B, Qwen3.6 35B-A3B, Qwen3.8 27B, GLM 4.7 Flash, Flint 40B-A2.5B (45 results).

What fits in 24 GB of VRAM?

By memory math at Q4_K_M (4.8944 bits per weight, llama.cpp), weights only, before KV cache and context: Ornith 1.5 35B-A3B (≈22 GB), Qwen3.6 35B-A3B (≈22 GB), Gemma 4 31B (≈19 GB), Nemotron 3.5 Lightning 30B-A3B (≈18 GB), Qwen3.6 27B (≈17 GB), Qwen3.8 27B (≈17 GB), Gemma 4 26B-A4B (≈15 GB), Devstral Small 2 24B (≈15 GB). Leave a few GB for context.

Are speculative-decoding results comparable?

Not 1:1. MTP, DFlash and similar methods draft tokens that the main model verifies, which can double tok/s on predictable text. This page lists them in a separate column; tick the box to hide them in the full table.

Not enough VRAM? Rent before you buy. Run bigger models on a rented 24–96 GB cloud GPU by the hour and see the real speed before spending €1,000+. Try it on Runpod
Referral link: if you sign up with Google and add $10, Runpod gives you and RigBase bonus credit (terms). It never affects our data.