A single 24GB GPU is the minimum needed for serious local language model inference. The memory is split into three parts: model weights, KV cache for context, and runtime overhead from the serving stack. At the common Q4_K_M quantization each parameter costs about 0.58 bytes, so a 30B dense model needs roughly 17‑18 GB for weights alone. Adding about 1‑2 GB for runtime and a modest KV cache leaves room for a few thousand tokens of context. Mixture‑of‑Experts models must be sized by total parameters because every expert stays resident even if only a few are active per token.
For a 24GB card the sweet spot is 20B‑35B class models quantized to Q4_K_M. Qwen3.6‑27B (dense) fits comfortably with ~16 GB weights, leaving ample headroom for context and making it a strong all‑around choice, especially for agentic coding. Qwen3.6‑35B‑A3B is a MoE with 35B total and ~3B active; it needs ~20 GB at Q4_K_M, a tight but viable fit when speed matters more than maximum size. Gemma 4 26B (MoE, 3.8B active) offers vision and multilingual support in a similar footprint. Mistral Small 3.2 24B is dense and the lightest at ~14 GB, letting you push context longer. gpt‑oss‑20b (MoE, 3.6B active) loads in ~14 GB and excels at structured reasoning. DeepSeek‑R1‑Distill‑Qwen‑32B is the tightest fit at ~18‑20 GB, providing visible chain‑of‑thought for deep reasoning.
Avoid trying to squeeze 70B quantizations; they exceed the budget even with aggressive quantization. Frontier MoE models like GLM‑5.2, Kimi K2.7, DeepSeek V4, or Mistral Large 3 require multiple GPUs or high‑memory servers and are not viable on a single 24GB card.
To run these models locally use Ollama for simplicity, llama.cpp for fine‑grained quantization control, or vLLM for higher throughput. Pick the model that matches your primary task, keep context lengths reasonable, and let the GPU do the work without sending data to the cloud.
#AI #Product #LLM #GPU #Inference #OpenSource