Many people mix up two different ideas when they look at model files: the container that holds the weights on disk and the quantization method that tells how those weights were squeezed into fewer bits. Understanding the difference solves most of the confusion.
Containers are just file formats. Safetensors and the older PyTorch .bin/.pt files store raw tensors. Safetensors is safe because it cannot run code. GGUF is another container made for llama.cpp; it can also hold the tokenizer and chat template in one file.
Quantization methods are the tricks that reduce the number of bits per weight. GPTQ, AWQ, bitsandbytes NF4, and the GGUF family of Q-types are examples. Some methods like EXL2 and EXL3 bundle a quantization scheme with a special layout that works only with their own inference library.
A quick rule for estimating weight memory is: parameters multiplied by bits per weight divided by eight. This gives the baseline size; add KV cache and runtime overhead on top.
When deciding what to use, think about your hardware and workflow. If you are on a Mac, CPU only, or need to run a model larger than your GPU memory, start with GGUF. Try Q4_K_M first; move to Q5_K_M or Q6_K if you have spare memory.
For serving many users on data‑center GPUs, AWQ or GPTQ in vLLM or SGLang gives good speed and accuracy, or use FP8 on Hopper/Blackwell cards if you have them.
If you are a single user with a consumer NVIDIA GPU and want the highest tokens per second, EXL3 via TabbyAPI is the current best choice; EXL2 works for older setups.
For budget fine‑tuning, load a 16‑bit model and let bitsandbytes NF4 quantize it on the fly with QLoRA.
On Apple Silicon with a Python workflow, MLX offers a simple safetensors based option that runs natively.
Pick the container that matches your runtime, then choose the quantization method that fits your bit‑rate and quality needs.
#AI #Product #LLM #Quantization #GGUF #GPTQ