What Quantization Does
A Llama 3.1 70B model in float32 takes 280 GB of RAM - impractical on any consumer hardware. Quantization reduces the precision of model weights from 32-bit or 16-bit floats to 8-bit or 4-bit integers. The result: dramatically lower memory requirements with a small, usually acceptable quality loss.
GGUF (GPT-Generated Unified Format) is the file format used by llama.cpp and tools like Ollama and LM Studio. It stores quantized weights with metadata about the quantization scheme.
Quantization Levels Explained
| Level | Bits | Method | Quality | Notes |
|---|---|---|---|---|
| Q4_0 | 4 | Simple block quant | Moderate | Fastest, lowest quality |
| Q4_K_M | 4 | K-quant, medium | Good | Best 4-bit for most use cases |
| Q4_K_S | 4 | K-quant, small | Moderate | Smaller than Q4_K_M |
| Q5_K_M | 5 | K-quant, medium | Very good | ~20% more RAM than Q4_K_M |
| Q6_K | 6 | K-quant | Near-lossless | Excellent for 13B and smaller |
| Q8_0 | 8 | 8-bit block quant | Near-perfect | 2x size of Q4, minimal quality loss |
| F16 | 16 | Half precision | Reference | ~2x Q8_0, baseline quality |
K-quant methods (K_M, K_S, K_L) use a smarter quantization scheme than naive Q4_0: they group weights into blocks and allocate higher precision to weights that matter more (typically attention layers). Q4_K_M is the community default for 4-bit inference.
